[{"content":"Lesson 1 — The Big Picture: What Is a Diffusion Model? Course map. Ten lessons. We start from intuition, build the forward (noising) process, derive the reverse (denoising) process from the variational lower bound, collapse it into the famous one-line training loss, study the network that implements it, write the training loop, then the sampling loop, then the fast/deterministic samplers and the score-based view, and finally conditioning, guidance and latent diffusion. Each lesson opens with a recap of the previous one and closes with a conclusion.\nCourse map and prerequisites. The full ten-lesson table of contents, the suggested reading order, and a prerequisites list (what background to have before starting, and where each piece is first needed) live in the Course Index . The complete symbol glossary is in Appendix B ; acronyms are in Appendix A . This lesson assumes only comfort with deep learning; the probability machinery enters in Lessons 2–4.\nDestruction is the easy art. A photograph left in the sun fades; a struck match burns down to nothing; a clear radio signal dissolves into hiss. None of it takes talent. The strange, almost suspicious feat would be to run any of these backwards — to coax a sharp image out of a fog of static, one careful step at a time, and have it emerge not as the original but as something wholly new and equally convincing. A diffusion model is a machine built to perform exactly that feat. This first lesson is the view from altitude, before we descend into the gears.\nOverview Imagine a sharp photograph that you corrupt by sprinkling sand on it, a little at a time. After one handful you can still see the picture; after a hundred it is structureless sand. This destruction is easy: anyone can pour sand. The hard, magical part is the reverse — removing exactly one handful and recovering the slightly cleaner image underneath. If you could do that one tiny step reliably, you could start from pure sand and, by repeating the small reversal, conjure a brand-new sharp photograph that never existed.\nThat is the entire idea:\nForward process (training-time only): take real data and gradually corrupt it with Gaussian noise until it becomes (approximately) $\\mathcal N(0,I)$. This direction is fixed, has no learnable parameters, and is trivial to evaluate. Reverse process (the learned model): train a neural network to undo one small step of corruption. At inference we start from $x_T\\sim\\mathcal N(0,I)$ and apply this learned denoiser many times until a clean $x_0$ emerges. The asymmetry is the whole trick. Adding noise is free; removing it is what we learn. Because each step removes only a tiny amount of noise, the conditional it must model is gentle and (as we will show in Lesson 3) close to Gaussian — which is precisely why diffusion models train stably and sample at high quality.\nFigure 1. The asymmetry the whole method rests on.\nWhere this sits in the generative-model landscape Before we touch a single equation, it helps to know what kind of animal we are dealing with — and the quickest way to know an animal is to stand it beside its relatives.\nGenerative modeling means learning $p_\\theta(x)$ that approximates the true data law $q(x)$ so we can draw novel $x\\sim p_\\theta$. Families differ along three axes: likelihood tractability (can we evaluate or bound $\\log p_\\theta(x)$?), sampling cost (how many network evaluations per sample?), and training stability (is the objective a well-posed optimization, or a min–max/adversarial one?).\nFamily Likelihood Sampling cost Training Failure mode GAN none (implicit) 1 forward pass (fast) min–max game non-convergence, mode collapse VAE ELBO lower bound 1 forward pass (fast) single max-likelihood bound (stable) blurry samples, posterior collapse Autoregressive exact $\\log p_\\theta$ $O(\\dim x)$ sequential (slow) exact MLE (stable) slow sampling, ordering bias Normalizing flow exact $\\log p_\\theta$ 1 pass (fast) exact MLE (stable) constrained invertible architectures; expensive Jacobians Diffusion ELBO lower bound $O(T)$ passes (slow; $T$ up to $10^3$) regression / ELBO (very stable) slow sampling (mitigated by DDIM, distillation) A few points sharpened:\nGANs optimize $\\min_\\theta\\max_\\phi V(\\theta,\\phi)$; sampling is a single generator pass, but there is no explicit density and the saddle-point objective is delicate (Goodfellow et al., 2014). VAEs maximize the ELBO $\\mathbb E_{q_\\phi(z\\mid x)}[\\log p_\\theta(x\\mid z)] - \\mathrm{KL}(q_\\phi(z\\mid x)\\,\\|\\,p(z))$ jointly over encoder $q_\\phi$ and decoder $p_\\theta$ (Kingma \u0026amp; Welling, 2014). Autoregressive models give exact likelihood via $p(x)=\\prod_i p(x_i\\mid x_{\\lt i})$ but require $O(\\dim x)$ sequential evaluations to sample. Normalizing flows use invertible $f_\\theta$ with $\\log p_\\theta(x)=\\log p(z)+\\log|\\det \\partial f_\\theta/\\partial x|$; exact likelihood at the cost of architectural invertibility and tractable log-determinants (Rezende \u0026amp; Mohamed, 2015). Diffusion turns generation into many small denoising steps; the objective is a stable regression (Sohl-Dickstein et al., 2015; Ho et al., 2020), at the cost of $O(T)$ network evaluations per sample. Diffusion as a hierarchical VAE with a fixed encoder The latent-variable view makes the \u0026ldquo;deeply hierarchical VAE\u0026rdquo; claim precise. A diffusion model defines a joint over data $x_0$ and a chain of latents $x_{1:T}$:\n$$ p_\\theta(x_{0:T}) = p(x_T)\\prod_{t=1}^{T} p_\\theta(x_{t-1}\\mid x_t), \\qquad p(x_T)=\\mathcal N(0,I). $$A standard VAE has an encoder $q_\\phi(z\\mid x)$ and a decoder $p_\\theta(x\\mid z)$, trained jointly. A diffusion model is the special case of a $T$-level hierarchical VAE with latents $z=(x_1,\\dots,x_T)$ in which the inference network is not learned but fixed to the forward chain\n$$ q(x_{1:T}\\mid x_0)=\\prod_{t=1}^T q(x_t\\mid x_{t-1}), \\qquad q(x_t\\mid x_{t-1})=\\mathcal N\\!\\big(\\sqrt{1-\\beta_t}\\,x_{t-1},\\,\\beta_t I\\big). $$Because $q$ has no parameters $\\phi$, only the decoder chain $p_\\theta(x_{t-1}\\mid x_t)$ is learned. The training objective is the same ELBO as a VAE — maximize a lower bound on $\\log p_\\theta(x_0)$ — and reduces, term by term, to making the learned reverse $p_\\theta$ match the true reverse of the fixed $q$. We derive that ELBO and its per-step decomposition in Lesson 4.\nRemark (why fix the encoder?). Freezing $q$ removes the encoder-posterior optimization that makes hierarchical VAEs hard to train, and it gives the forward marginals $q(x_t\\mid x_0)$ in closed form (Lesson 2). The price is that the latents share the dimensionality of the data and the chain is long, hence the $O(T)$ sampling cost.\nFigure 2. Five families on three axes.\nThe two distributions you must keep separate If you carry one thing out of this lesson and into the next nine, carry this: there are two processes in the room, and mistaking one for the other is the surest way to get lost.\nThroughout the course, hold these apart:\n$q(\\cdot)$ — the forward / noising process. Fixed. Known in closed form. No parameters. $p_\\theta(\\cdot)$ — the reverse / denoising process. Learned. Parameterized by a neural network with weights $\\theta$. The objective is, in essence, make $p_\\theta$ agree with the true reverse of $q$. Almost every equation in this course refines that one sentence.\nNotation we will use $x_0$ — a real data point (e.g. an image), drawn from $q(x_0)$. $x_1,\\dots,x_T$ — increasingly noisy latents; $x_T$ is essentially pure noise. $T$ — number of diffusion steps (often $1000$ for training). $\\beta_t\\in(0,1)$ — the variance schedule; $\\alpha_t:=1-\\beta_t$ and $\\bar\\alpha_t:=\\prod_{s\\le t}\\alpha_s$. $a_t:=\\sqrt{\\bar\\alpha_t}$, $b_t:=\\sqrt{1-\\bar\\alpha_t}$ — the signal and noise scales (so $x_t=a_t x_0+b_t\\epsilon$ and $a_t^2+b_t^2=1$). $\\epsilon\\sim\\mathcal N(0,I)$ — standard Gaussian noise; $\\epsilon_\\theta$ — the network\u0026rsquo;s noise predictor. $\\mu_\\theta,\\Sigma_\\theta$ — learned reverse mean/covariance; $\\tilde\\mu_t,\\tilde\\beta_t$ — true posterior mean/variance (Lesson 3); $\\sigma_t$ — sampling variance (Lesson 8). $\\hat x_0$ — the network\u0026rsquo;s implied estimate of the clean image; $v$ — the velocity target (Lesson 5); $\\mathrm{SNR}(t)=\\bar\\alpha_t/(1-\\bar\\alpha_t)$ — signal-to-noise ratio (Lesson 2); $c$ — a conditioning signal (Lesson 10). The forward chain runs $x_0\\to\\dots\\to x_T$ (clean to noise); the reverse runs $x_T\\to\\dots\\to x_0$. Training learns the reverse; sampling runs it.\nNotation strategy. New symbols are introduced where they first do work (e.g. $\\tilde\\beta_t$ in Lesson 3, $\\sigma_t$ in Lesson 8), so a non-linear reader may meet a symbol before its definition. The symbol glossary in Appendix B gathers every symbol with its meaning in one place — treat it as the canonical lookup table for the whole course.\nA first glimpse of the math (no derivation yet) A single forward step adds a controlled amount of Gaussian noise:\n$$ q(x_t\\mid x_{t-1}) = \\mathcal N\\!\\left(x_t;\\ \\sqrt{1-\\beta_t}\\,x_{t-1},\\ \\beta_t I\\right). $$The next, noisier image is the previous one shrunk slightly toward zero ($\\sqrt{1-\\beta_t}\\lt 1$) plus fresh Gaussian noise of variance $\\beta_t$. The contraction keeps the variance bounded, so after many steps we land on a unit Gaussian. Lesson 2 dissects this.\nThe model approximates the reverse step, which we assume is also Gaussian:\n$$ p_\\theta(x_{t-1}\\mid x_t) = \\mathcal N\\!\\left(x_{t-1};\\ \\mu_\\theta(x_t,t),\\ \\Sigma_\\theta(x_t,t)\\right). $$ Remark (when is the reverse Gaussian?). Feller\u0026rsquo;s classical result on diffusion processes implies that for sufficiently small step size $\\beta_t$, the true reverse conditional $q(x_{t-1}\\mid x_t)$ is itself approximately Gaussian. This is why $\\beta_t$ must be small and $T$ large, and it is the conceptual cornerstone we return to in Lesson 3.\nFigure 3. Two distributions in the room, never to be confused.\nExercises Try these before reading on; they check the conceptual core of this lesson. Solutions are in the collapsible blocks.\n1. Why freeze the encoder? A standard VAE learns its encoder $q_\\phi(z\\mid x)$. A diffusion model instead fixes the \u0026ldquo;encoder\u0026rdquo; to the noising chain $q(x_{1:T}\\mid x_0)$. State two concrete consequences of this choice — one that makes diffusion easier than a hierarchical VAE, and one that is the price paid.\nEasier: because $q$ has no parameters $\\phi$, there is nothing to optimize on the inference side — we avoid the encoder-posterior optimization that destabilizes deep hierarchical VAEs, and (Lesson 2) the forward marginals $q(x_t\\mid x_0)$ are available in closed form, so we can jump to any noise level in one operation. Price: the latents $x_{1:T}$ share the full dimensionality of the data and the chain is long ($T$ up to $10^3$), giving the $O(T)$ sampling cost.\n2. Reading the landscape table. Which generative family offers both exact log-likelihood and single-pass sampling, and what does it pay for that? What does diffusion give up by comparison, and what does it buy?\nNormalizing flows give exact $\\log p_\\theta$ and one-pass sampling, paying with architectural invertibility constraints and expensive Jacobian log-determinants. Diffusion gives up exact likelihood (only an ELBO) and fast sampling ($O(T)$ passes), but buys a stable, well-posed regression objective (no min–max game) and high sample quality.\n3. The intractable marginal. Write the integral that defines $p_\\theta(x_0)$ from the reverse joint $p_\\theta(x_{0:T})$, and say in one sentence why it is intractable (hence why we need the bound of Lesson 4).\n$$ p_\\theta(x_0)=\\int p_\\theta(x_{0:T})\\,dx_{1:T}=\\int p(x_T)\\prod_{t=1}^{T}p_\\theta(x_{t-1}\\mid x_t)\\,dx_{1:T}. $$It is a high-dimensional integral over every latent trajectory $x_{1:T}$ with no closed form, so $\\log p_\\theta(x_0)$ cannot be evaluated directly — motivating the variational lower bound.\nConclusion A diffusion model rests on a deliberate asymmetry: a fixed, parameter-free forward process that drowns data in noise, and a learned reverse process that removes noise one gentle step at a time. Formally it is a hierarchical latent-variable model — a VAE whose encoder is frozen to the noising chain — trained by the same ELBO machinery but with a stable regression-like objective. We carry forward two characters, the fixed $q$ and the learned $p_\\theta$, and one mission: make the learned reverse match the true reverse of the noising process. In Lesson 2 we make the forward process concrete and uncover the single most useful property in all of diffusion modeling: the closed-form jump to any noise level $t$.\nThat is the altitude view. The wish we opened with — to run destruction backwards and have something new fall out — is no longer a daydream but a plan with a shape. Lesson 2 begins where it must: by understanding the destruction itself, in enough detail that we will later dare to reverse it.\nReferences Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., \u0026amp; Ganguli, S. (2015). Deep Unsupervised Learning using Nonequilibrium Thermodynamics. ICML. Ho, J., Jain, A., \u0026amp; Abbeel, P. (2020). Denoising Diffusion Probabilistic Models (DDPM). NeurIPS. Kingma, D. P., \u0026amp; Welling, M. (2014). Auto-Encoding Variational Bayes (VAE). ICLR. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., \u0026amp; Bengio, Y. (2014). Generative Adversarial Nets. NeurIPS. Rezende, D. J., \u0026amp; Mohamed, S. (2015). Variational Inference with Normalizing Flows. ICML. Yang, L., Zhang, Z., Song, Y., et al. (2023). Diffusion Models: A Comprehensive Survey of Methods and Applications. ACM Computing Surveys. Course Index · Next: Lesson 2 — The Forward Process → · Appendix A: Acronyms · Appendix B: Formulas The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/tutorials/diffusion-lessons/lesson-01-the-big-picture/","summary":"\u003ch1 id=\"lesson-1--the-big-picture-what-is-a-diffusion-model\"\u003eLesson 1 — The Big Picture: What Is a Diffusion Model?\u003c/h1\u003e\n\u003cblockquote\u003e\n\u003cp\u003e\u003cstrong\u003eCourse map.\u003c/strong\u003e Ten lessons. We start from intuition, build the \u003cem\u003eforward\u003c/em\u003e (noising) process, derive the \u003cem\u003ereverse\u003c/em\u003e (denoising) process from the variational lower bound, collapse it into the famous one-line training loss, study the network that implements it, write the training loop, then the sampling loop, then the fast/deterministic samplers and the score-based view, and finally conditioning, guidance and latent diffusion. Each lesson opens with a recap of the previous one and closes with a conclusion.\u003c/p\u003e","title":"The Big Picture"},{"content":"Lesson 1 — The Big Picture: Training vs. Inference A 10-lesson technical course on the mechanics of large language models. Audience: technically comfortable, new to ML. Prerequisites: linear algebra (matrix products, vector spaces), multivariate calculus (gradients, the chain rule), and basic probability (distributions, expectation). We use standard notation throughout and define it as we go.\nCourse table of contents # Lesson Core technical content 1 The Big Picture (this file) The map $f(x;\\theta)$; training vs. inference; parameters vs. config vs. optimizer state; FLOP and memory accounting 2 Tokenization \u0026amp; Embeddings BPE; the embedding map $E\\in\\mathbb R^{V\\times d}$; one-hot lookups; sinusoidal vs. learned vs. rotary (RoPE) positions 3 Self-Attention $\\mathrm{softmax}(QK^\\top/\\sqrt{d_k})V$; the $\\sqrt{d_k}$ variance argument; softmax Jacobian; causal masking; multi-head; $O(n^2d)$ cost 4 The Transformer Block FFN and parameter budget; GELU/SwiGLU; residual gradient flow; LayerNorm forward + backward; RMSNorm; pre- vs. post-norm 5 Output Head \u0026amp; Objective Unembedding $W_U$; weight tying; the autoregressive factorization; the $\\partial\\mathcal L/\\partial \\mathbf z=\\hat{\\mathbf p}-\\mathbf y$ identity 6 The Loss Function Cross-entropy as $H(p)+D_{\\mathrm{KL}}(p\\|\\hat p)$; nats vs. bits; perplexity; logit-gradient derivation 7 Backprop \u0026amp; Gradient Descent Reverse-mode autodiff; VJPs; per-layer adjoint equations; activation checkpointing; vanishing/exploding gradients 8 Optimizers, LR \u0026amp; Batches SGD→momentum→Adam with bias-correction derivation; AdamW; clipping; mixed precision; scaling laws 9 Fine-Tuning, SFT \u0026amp; RLHF SFT loss masking; LoRA $W+\\frac{\\alpha}{r}BA$; Bradley–Terry reward; PPO clipped objective; DPO loss 10 Inference in Depth Greedy/beam/sampling; temperature, top-k, top-p; KV-cache memory; GQA/MQA; speculative decoding A Appendix A — Acronyms Every acronym and symbol used in the course, expanded and explained in plain language B Appendix B — Formulas All key equations, collected by lesson with a one-line reminder of what each says Two invariants run through every lesson and are worth stating before anything else:\nTraining writes; inference reads. The forward pass is shared by both phases; only training additionally runs the backward pass and the parameter update. Three kinds of numbers, never to be confused: parameters (learned), configuration (chosen), optimizer/runtime state (transient). 1. The model as a parameterized function In plain terms: a language model is a function that reads the text so far and outputs a probability for every word in its vocabulary as the next word — a guess like \u0026ldquo;70% chance the next token is the, 5% a, …\u0026rdquo; with the probabilities summing to 1. The set of all such valid probability lists (non-negative, summing to one) is called the probability simplex; that is all the symbol $\\Delta^{V-1}$ below denotes. With that picture in mind, the formal statement is:\n$$ f:\\ \\mathcal V^{\\le n}\\times\\Theta \\to \\Delta^{V-1},\\qquad \\hat{\\mathbf p}=f(x;\\theta), $$mapping a token sequence $x=(x_1,\\dots,x_t)$ drawn from a vocabulary $\\mathcal V$ of size $V$, together with a parameter vector $\\theta\\in\\Theta=\\mathbb R^{P}$, to a point on the $(V{-}1)$-dimensional probability simplex $\\Delta^{V-1}$ — i.e. a categorical distribution over the next token. (It is \u0026ldquo;$V-1$\u0026quot;-dimensional because once $V-1$ of the probabilities are fixed, the last is forced by the sum-to-one constraint.) \u0026ldquo;$P$ parameters\u0026rdquo; literally means $\\theta$ is a vector in $\\mathbb R^P$; for frontier models $P$ ranges from $10^9$ to $10^{12}$.\nThe internal structure of $f$ — embeddings, attention, MLPs, normalization, an output projection — is the subject of Lessons 2–5. For now treat $f$ as a black box with a known input/output type and a very high-dimensional parameter vector. Before we open that box, though, one detail of the output type does real work: $f$ predicts only a single next token, yet we speak of it modeling whole sequences. The next subsection shows why those are the same thing.\nThe autoregressive factorization Crucially $f$ models only the next token. A full sequence is scored by the chain rule of probability:\n$$ P_\\theta(x_1,\\dots,x_n)=\\prod_{t=1}^{n} P_\\theta(x_t\\mid x_{\\lt t}),\\qquad x_{\\lt t}:=(x_1,\\dots,x_{t-1}). $$Each factor $P_\\theta(x_t\\mid x_{\\lt t})$ is exactly one evaluation of $f$. This factorization is why next-token prediction is sufficient to model language in full generality — it is not an approximation but an identity — and it is the formal basis for both the training loss (Lesson 6) and autoregressive generation (Lesson 10).\nWith the object defined, the rest of the story is about two things we do with it. We have to first build the parameter vector $\\theta$ that makes $f$ good, then run $f$ to produce text. These are not the same activity, and keeping them apart is the single most clarifying move in the whole course.\n2. The two phases, defined operationally Training: the optimization loop Given a corpus $\\mathcal D$, training solves the empirical risk minimization problem\n$$ \\theta^\\star=\\arg\\min_{\\theta\\in\\mathbb R^P}\\ \\mathcal L(\\theta),\\qquad \\mathcal L(\\theta)=\\mathbb E_{x\\sim\\mathcal D}\\!\\left[-\\sum_{t} \\log P_\\theta(x_t\\mid x_{\\lt t})\\right]. $$We cannot solve this in closed form — $f$ is wildly nonconvex in $\\theta$ — so we iterate first-order updates. One step consists of four operations on a mini-batch $B\\subset\\mathcal D$:\nForward pass: evaluate $\\hat{\\mathbf p}=f(x;\\theta)$ and the loss $\\mathcal L_B(\\theta)$, caching intermediate activations. Backward pass: compute the gradient $g=\\nabla_\\theta\\mathcal L_B$ by reverse-mode autodiff (Lesson 7). Optimizer step: map $g$ to a parameter update via the optimizer\u0026rsquo;s rule (Lesson 8), e.g. Adam. Apply: $\\theta \\leftarrow \\theta - (\\text{update})$. The loop is worth seeing as a picture — the forward pass produces the loss, the backward pass turns it into a gradient, and the optimizer writes that back into $\\theta$, then it all repeats:\nflowchart LR theta[\"θ (parameters)\"] --\u003e fwd[\"1 · Forward passp̂ = f(x; θ), compute loss L\"] fwd --\u003e|\"cached activations\"| bwd[\"2 · Backward passg = ∇θ L (Lesson 7)\"] bwd --\u003e opt[\"3 · Optimizer stepmap g → update (Lesson 8)\"] opt --\u003e apply[\"4 · Applyθ ← θ − update\"] apply --\u003e|\"next mini-batch\"| theta This loop runs for $10^5$–$10^6$ steps over $10^{12}$+ tokens. Training is the only regime in which $\\theta$ changes.\nInference: the forward pass alone At inference we evaluate only step 1, and only at the final position, to obtain $\\hat{\\mathbf p}=f(x;\\theta)$; we draw a token, append it, and repeat (Lesson 10). No loss, no gradient, no update. $\\theta$ is bit-for-bit constant. Any apparent \u0026ldquo;memory\u0026rdquo; of a conversation is the prior tokens re-entering through the context window $x$, not a change in $\\theta$.\nInvariant. Training is a map $\\theta_t \\mapsto \\theta_{t+1}$. Inference is a map $x\\mapsto \\hat{\\mathbf p}$ at fixed $\\theta$. Conflating them is the most common conceptual error about LLMs.\nFigure 1. One training step, and the loop it sits in.\n3. Three kinds of numbers The training/inference split already separated two activities. It pays to also separate the numbers those activities touch, because every quantity in the rest of the course belongs to exactly one of three categories — and most confusion about LLMs dissolves once you separate them cleanly.\n(a) Parameters $\\theta$ — learned. The entries of every weight matrix: the embedding $E$, the attention projections $W_Q,W_K,W_V,W_O$, the MLP weights $W_1,W_2$ and biases, the LayerNorm gains/shifts $\\gamma,\\beta$, the unembedding $W_U$. Initialized from a random distribution, updated by step 4 every iteration. These are the model. Tagged [LEARNED] throughout.\n(b) Configuration / hyperparameters — chosen. Two sub-families:\nArchitectural: depth $N$, width $d$, head count $h$, head dim $d_k$, MLP width $d_{ff}$, vocab $V$, context length $n$. These determine the shape of $\\theta$ (and hence $P$) but are not entries of $\\theta$. Optimization \u0026amp; inference: learning rate $\\eta$ and its schedule, batch size, weight decay $\\lambda$, Adam\u0026rsquo;s $(\\beta_1,\\beta_2,\\epsilon)$, dropout, and at inference temperature/top-k/top-p. These govern the process, not the model. Tagged [CONFIG]. (c) Optimizer \u0026amp; runtime state — transient. Adam\u0026rsquo;s first/second moment estimates $(\\mathbf m,\\mathbf v)\\in\\mathbb R^P\\times\\mathbb R^P$, the cached forward activations needed for backprop, the KV cache at inference, and every logit/probability vector. None of these are part of the deployed model; all are recomputed or discarded. Tagged [STATE].\nProperty Parameters $\\theta$ Configuration Optimizer/runtime state Set by gradient updates humans / search computed during a run Changes in training? yes (this is training) no yes (e.g. $\\mathbf m,\\mathbf v$) Persists into deployed model? yes yes (defines/configures it) no Count (order of magnitude) $10^9$–$10^{12}$ $10^1$–$10^2$ $\\sim 2P$ (Adam) + activations A crisp operational test: if step 4 of the loop writes to it, it is a parameter; if a human fixes it outside the loop, it is configuration; if it exists only to make a run go and is then thrown away, it is state.\nFigure 2. Three kinds of numbers, never to be confused.\n4. Quantifying the two phases: FLOPs and memory So far the training/inference distinction has been qualitative. Putting numbers on it makes it concrete — and the numbers turn out to explain, in one stroke, why building a model and running it sit at such different scales. Being concrete about cost sharpens the training/inference distinction.\nCompute. A standard estimate for a dense transformer is that one forward pass costs $\\approx 2P$ floating-point operations per token (each parameter participates in roughly one multiply-add, and a multiply-add is 2 FLOPs). The backward pass costs about twice the forward pass (it computes gradients with respect to both inputs and parameters), so a full training step is $\\approx 6P$ FLOPs/token. Hence the well-known rule of thumb for total pretraining compute:\n$$ C \\approx 6\\,P\\,D \\ \\text{FLOPs}, $$where $D$ is the number of training tokens. Inference, by contrast, is $\\approx 2P$ FLOPs/token — the missing factor of 3 is precisely the backward pass and is exactly why training a model costs orders of magnitude more than running it.\nMemory. Training must hold, per parameter (in a mixed-precision Adam setup): the parameter itself, its gradient, and two optimizer moments — plus a master fp32 copy — totaling on the order of 16 bytes/parameter, before counting cached activations (which scale with batch size × sequence length × depth). Inference holds only the parameters (often quantized to 1–2 bytes each) plus the KV cache (Lesson 10). This is why a model that fits comfortably on one accelerator for inference may require a large cluster to train.\nTakeaway. \u0026ldquo;Training is expensive, inference is cheap(er)\u0026rdquo; is not vibes — it is the $6PD$ vs. $2P$-per-token gap, plus the $\\sim$4× memory overhead of gradients and optimizer state. Inference carries none of that.\nFigure 3. Why writing the model costs so much more than reading it.\n5. The mental model, restated precisely Picture $\\theta$ as a point in $\\mathbb R^P$ and $\\mathcal L:\\mathbb R^P\\to\\mathbb R$ as a scalar field (the loss landscape) over it. Training is iterative descent on this field — each step reads the local gradient and moves $\\theta$ downhill (Lesson 7). Configuration fixes the dimension and shape of the space ($P$ itself) and the rules of descent (step size, momentum). Inference evaluates the fixed function $f(\\cdot;\\theta^\\star)$; the only freedom left is how we sample from its output distribution, which perturbs the output without touching the point $\\theta^\\star$.\nConclusion A large language model is a parameterized function $f(x;\\theta)$ that outputs a categorical distribution over the next token; the chain-rule factorization $P_\\theta(x_{1:n})=\\prod_t P_\\theta(x_t\\mid x_{\\lt t})$ makes next-token prediction a complete model of sequences. Training solves $\\theta^\\star=\\arg\\min_\\theta\\mathcal L(\\theta)$ by an iterative forward/backward/update loop and is the only phase that writes to $\\theta$; inference evaluates the forward pass at frozen $\\theta$. Three categories of numbers must be kept distinct: parameters (learned, $\\sim$$P$ of them), configuration (chosen, defines shape and process), and optimizer/runtime state (transient, e.g. Adam moments and the KV cache). The $6PD$-vs-$2P$ compute gap and the $\\sim$4× memory overhead of training quantify why writing the model is so much costlier than reading it.\nThe next nine lessons open the box $f$, derive the loss and its gradient, and specify each update rule — always tagging quantities as [LEARNED], [CONFIG], or [STATE]. We open the box from the bottom up, and the bottom is the boundary where messy human text first becomes the numbers $f$ can act on. Lesson 2 builds that boundary: how a string turns into the input tensor $H^{(0)}$.\n← Previous: (none — this is the first lesson) · Course index · Next: Lesson 2 — Tokenization \u0026amp; Embeddings → Appendices: A — Acronyms · B — Formulas · C — Attention Variants The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/tutorials/llm-lessons/lesson-01-the-big-picture/","summary":"\u003ch1 id=\"lesson-1--the-big-picture-training-vs-inference\"\u003eLesson 1 — The Big Picture: Training vs. Inference\u003c/h1\u003e\n\u003cp\u003e\u003cem\u003eA 10-lesson technical course on the mechanics of large language models.\u003c/em\u003e\n\u003cem\u003eAudience: technically comfortable, new to ML. Prerequisites: linear algebra (matrix products, vector spaces), multivariate calculus (gradients, the chain rule), and basic probability (distributions, expectation). We use standard notation throughout and define it as we go.\u003c/em\u003e\u003c/p\u003e\n\u003chr\u003e\n\u003ch2 id=\"course-table-of-contents\"\u003eCourse table of contents\u003c/h2\u003e\n\u003ctable\u003e\n  \u003cthead\u003e\n      \u003ctr\u003e\n          \u003cth\u003e#\u003c/th\u003e\n          \u003cth\u003eLesson\u003c/th\u003e\n          \u003cth\u003eCore technical content\u003c/th\u003e\n      \u003c/tr\u003e\n  \u003c/thead\u003e\n  \u003ctbody\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e1\u003c/td\u003e\n          \u003ctd\u003e\u003cstrong\u003eThe Big Picture\u003c/strong\u003e \u003cem\u003e(this file)\u003c/em\u003e\u003c/td\u003e\n          \u003ctd\u003eThe map $f(x;\\theta)$; training vs. inference; parameters vs. config vs. optimizer state; FLOP and memory accounting\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e2\u003c/td\u003e\n          \u003ctd\u003e\u003ca href=\"/tutorials/llm-lessons/lesson-02-tokenization-and-embeddings/\"\u003eTokenization \u0026amp; Embeddings\u003c/a\u003e\n\u003c/td\u003e\n          \u003ctd\u003eBPE; the embedding map $E\\in\\mathbb R^{V\\times d}$; one-hot lookups; sinusoidal vs. learned vs. rotary (RoPE) positions\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e3\u003c/td\u003e\n          \u003ctd\u003e\u003ca href=\"/tutorials/llm-lessons/lesson-03-self-attention/\"\u003eSelf-Attention\u003c/a\u003e\n\u003c/td\u003e\n          \u003ctd\u003e$\\mathrm{softmax}(QK^\\top/\\sqrt{d_k})V$; the $\\sqrt{d_k}$ variance argument; softmax Jacobian; causal masking; multi-head; $O(n^2d)$ cost\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e4\u003c/td\u003e\n          \u003ctd\u003e\u003ca href=\"/tutorials/llm-lessons/lesson-04-the-transformer-block/\"\u003eThe Transformer Block\u003c/a\u003e\n\u003c/td\u003e\n          \u003ctd\u003eFFN and parameter budget; GELU/SwiGLU; residual gradient flow; LayerNorm forward + backward; RMSNorm; pre- vs. post-norm\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e5\u003c/td\u003e\n          \u003ctd\u003e\u003ca href=\"/tutorials/llm-lessons/lesson-05-output-head-and-objective/\"\u003eOutput Head \u0026amp; Objective\u003c/a\u003e\n\u003c/td\u003e\n          \u003ctd\u003eUnembedding $W_U$; weight tying; the autoregressive factorization; the $\\partial\\mathcal L/\\partial \\mathbf z=\\hat{\\mathbf p}-\\mathbf y$ identity\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e6\u003c/td\u003e\n          \u003ctd\u003e\u003ca href=\"/tutorials/llm-lessons/lesson-06-the-loss-function/\"\u003eThe Loss Function\u003c/a\u003e\n\u003c/td\u003e\n          \u003ctd\u003eCross-entropy as $H(p)+D_{\\mathrm{KL}}(p\\|\\hat p)$; nats vs. bits; perplexity; logit-gradient derivation\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e7\u003c/td\u003e\n          \u003ctd\u003e\u003ca href=\"/tutorials/llm-lessons/lesson-07-backpropagation-and-gradient-descent/\"\u003eBackprop \u0026amp; Gradient Descent\u003c/a\u003e\n\u003c/td\u003e\n          \u003ctd\u003eReverse-mode autodiff; VJPs; per-layer adjoint equations; activation checkpointing; vanishing/exploding gradients\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e8\u003c/td\u003e\n          \u003ctd\u003e\u003ca href=\"/tutorials/llm-lessons/lesson-08-optimizers-learning-rate-batches/\"\u003eOptimizers, LR \u0026amp; Batches\u003c/a\u003e\n\u003c/td\u003e\n          \u003ctd\u003eSGD→momentum→Adam with bias-correction derivation; AdamW; clipping; mixed precision; scaling laws\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e9\u003c/td\u003e\n          \u003ctd\u003e\u003ca href=\"/tutorials/llm-lessons/lesson-09-finetuning-sft-rlhf/\"\u003eFine-Tuning, SFT \u0026amp; RLHF\u003c/a\u003e\n\u003c/td\u003e\n          \u003ctd\u003eSFT loss masking; LoRA $W+\\frac{\\alpha}{r}BA$; Bradley–Terry reward; PPO clipped objective; DPO loss\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e10\u003c/td\u003e\n          \u003ctd\u003e\u003ca href=\"/tutorials/llm-lessons/lesson-10-inference-in-depth/\"\u003eInference in Depth\u003c/a\u003e\n\u003c/td\u003e\n          \u003ctd\u003eGreedy/beam/sampling; temperature, top-k, top-p; KV-cache memory; GQA/MQA; speculative decoding\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eA\u003c/td\u003e\n          \u003ctd\u003e\u003ca href=\"/tutorials/llm-lessons/appendix-a-acronyms/\"\u003eAppendix A — Acronyms\u003c/a\u003e\n\u003c/td\u003e\n          \u003ctd\u003eEvery acronym and symbol used in the course, expanded and explained in plain language\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eB\u003c/td\u003e\n          \u003ctd\u003e\u003ca href=\"/tutorials/llm-lessons/appendix-b-formulas/\"\u003eAppendix B — Formulas\u003c/a\u003e\n\u003c/td\u003e\n          \u003ctd\u003eAll key equations, collected by lesson with a one-line reminder of what each says\u003c/td\u003e\n      \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp\u003eTwo invariants run through every lesson and are worth stating before anything else:\u003c/p\u003e","title":"The Big Picture"},{"content":"Lesson 1 — The Big Picture: Sparse vs. Dense A 10-lesson technical course on the mechanics of Mixture-of-Experts language models. Audience: undergraduate level, already comfortable with dense decoder-only LLMs (you know next-token prediction, attention, the feed-forward network, and fine-tuning). Prerequisites: linear algebra (matrix products), multivariate calculus (gradients, the chain rule), basic probability. We use standard notation and define each new symbol as it appears.\nCourse table of contents # Lesson Core technical content 1 The Big Picture (this file) Conditional computation; $P_{\\text{tot}}$ vs. $P_{\\text{act}}$; compute $\\approx 6P_{\\text{act}}D$ but memory $\\propto P_{\\text{tot}}$; the dense contrast 2 From the Dense FFN to a Layer of Experts The FFN holds $\\tfrac23$ of params; $E$ replicated experts; the MoE-layer map; placement; shared / fine-grained experts 3 The Router / Gating Network $g(\\mathbf x)=\\mathrm{softmax}(W_g\\mathbf x)$; top-$k$; noisy top-$k$; softmax-before vs. -after; the only new learned mechanism 4 Sparse Top-k Computation \u0026amp; the Combine The weighted sum $\\sum_{i\\in\\mathcal T_k}g_i E_i(\\mathbf x)$; sparse compute, dense memory; gradient through a discrete choice 5 Load Balancing \u0026amp; Auxiliary Losses Expert collapse; importance/load loss; Switch loss $\\alpha E\\sum_i f_iP_i$; z-loss; aux-loss-free bias 6 Capacity \u0026amp; Token Dropping Capacity factor $C_f$; capacity $=C_fkT/E$; overflow/dropping; dispatch \u0026amp; combine tensors 7 Expert Parallelism Sharding experts; the two all-to-all collectives; communication-bound training; GShard 8 Training Dynamics \u0026amp; Scaling Router instability \u0026amp; z-loss; router precision; emergent specialization; params-at-fixed-FLOPs scaling 9 Fine-Tuning MoE Overfitting of sparse models; expert dropout; selective FT; router freezing; vs. dense SFT/RLHF 10 Inference in Depth Memory-vs-compute asymmetry; batch routing \u0026amp; load imbalance; unchanged KV cache; offloading \u0026amp; serving A Appendix A — MoE Family Shazeer, GShard, Switch, GLaM, Mixtral, DeepSeek-MoE along the axes that differ B Appendix B — Dense vs. MoE Every contrast in one decision-oriented table C Appendix C — Acronyms Every acronym and symbol, expanded Two invariants run through every lesson and are worth stating before anything else:\nSparse activation: all of $\\theta$ is stored and trained, but only a $k/E$ slice of the experts is computed per token. This single fact splits every cost in the course into two columns — one that scales with total parameters, one that scales with activated parameters. A free router collapses. Left unconstrained, the router learns to send almost every token to a handful of experts; MoE training therefore always carries an explicit balancing pressure that has no analogue in a dense model. This course assumes the companion dense course as background. We do not re-derive attention, the loss, backprop, or the optimizer — those are unchanged. We change exactly one component of the transformer block and trace the consequences of that change all the way through training and inference.\n1. The dense model, in one paragraph of recap A dense decoder-only LLM is a parameterized function $f(x;\\theta)$ mapping a token sequence to a categorical distribution over the next token, $\\hat{\\mathbf p}=f(x;\\theta)\\in\\Delta^{V-1}$, and it scores whole sequences by the autoregressive chain rule $P_\\theta(x_{1:n})=\\prod_t P_\\theta(x_t\\mid x_{\\lt t})$. Its body is a stack of $N$ transformer blocks, each a multi-head attention sub-layer and a position-wise feed-forward network (FFN), wrapped in residual connections and normalization. The decisive structural fact for this course — established in the dense course\u0026rsquo;s Lesson 4 — is that the FFN holds about two-thirds of the model\u0026rsquo;s parameters, and that every parameter, FFN included, is multiplied through for every token. A dense forward pass costs $\\approx 2P$ FLOPs/token and a training step $\\approx 6P$; total pretraining compute is $C\\approx 6PD$ for $D$ tokens. Crucially, in a dense model there is only one parameter count $P$: it is simultaneously the model\u0026rsquo;s capacity and its per-token cost. Those two meanings are about to come apart.\n2. Conditional computation: the one idea The motivating observation is simple. A dense FFN applies the same transformation to every token, even though most tokens plausibly need only a small, specialized part of the model\u0026rsquo;s knowledge — the part that handles code, or French, or arithmetic, or biochemistry. Forcing each token through the entire FFN is wasteful: we pay for the whole network to process a token that needs a sliver of it.\nConditional computation is the idea of making the computation depend on the input — activating different parameters for different tokens. A Mixture-of-Experts layer is the dominant realization of that idea in modern LLMs. It replaces the single FFN of a block with:\na set of $E$ experts $\\{E_1,\\dots,E_E\\}$, each an independent FFN with its own weights ([LEARNED]), and a small router (or gate) $g$ that, for each token, produces a sparse weighting over the $E$ experts and selects the top $k$ of them ([LEARNED], but tiny). The layer\u0026rsquo;s output for a token $\\mathbf x$ is a weighted sum over only the selected experts:\n$$ \\mathrm{MoE}(\\mathbf x)=\\sum_{i\\in\\mathcal T_k(\\mathbf x)} g_i(\\mathbf x)\\,E_i(\\mathbf x), $$where $\\mathcal T_k(\\mathbf x)\\subseteq\\{1,\\dots,E\\}$ is the set of top-$k$ expert indices for this token and $g_i(\\mathbf x)$ is the gate weight assigned to expert $i$. Because $|\\mathcal T_k|=k$ and typically $k\\in\\{1,2\\}$ while $E$ ranges from $8$ to several hundred, each token is processed by only a $k/E$ fraction of the expert parameters. The other experts are not evaluated for this token at all — their FLOPs are simply never spent.\nThis is the entire architectural idea. Everything else in the course — the router, the balancing losses, the capacity factor, the all-to-all communication, the inference asymmetry — is a consequence of making this one sum work at scale, in training and at serving time. We build the layer itself in Lessons 2–4; here we establish why it changes the accounting.\nContrast with the dense model. A dense block computes $\\mathrm{FFN}(\\mathbf x)$ — one function, all parameters, every token. An MoE block computes $\\sum_{i\\in\\mathcal T_k(\\mathbf x)}g_i(\\mathbf x)E_i(\\mathbf x)$ — a token-chosen subset of $k$ out of $E$ functions. The dense model has one execution path; the MoE has $\\binom{E}{k}$ possible paths, and the router picks one per token.\nFigure 1. One component changed: the FFN becomes a routed committee.\n3. The two parameter counts The moment computation became conditional, the single number $P$ split into two, and keeping them apart is the single most clarifying move in this course — the direct analogue of the dense course\u0026rsquo;s \u0026ldquo;training writes, inference reads.\u0026rdquo;\nTotal parameters $P_{\\text{tot}}$ — what the model is. The count of every weight that must be stored and trained: attention, the router, and all $E$ experts in every MoE layer. This is the model\u0026rsquo;s capacity — how much it can know. It is what sets the memory footprint and what people mean when they call a model \u0026ldquo;a 47B-parameter Mixtral\u0026rdquo; or \u0026ldquo;a 671B-parameter DeepSeek-V3.\u0026rdquo;\nActivated parameters $P_{\\text{act}}$ — what the model does per token. The count of weights actually multiplied for a given token: attention (all of it, always), the router (tiny), and only the $k$ selected experts. This is the per-token compute cost — what sets FLOPs and, in large part, latency. It is what people mean when they say Mixtral-8×7B \u0026ldquo;activates about 13B parameters\u0026rdquo; or DeepSeek-V3 \u0026ldquo;activates about 37B of its 671B.\u0026rdquo;\nFor a model whose MoE layers dominate the parameter budget, the ratio is approximately\n$$ \\frac{P_{\\text{act}}}{P_{\\text{tot}}}\\approx\\frac{k}{E}\\quad\\text{(over the expert parameters)}, $$so $k=2,\\,E=8$ activates roughly a quarter of the experts, and $k=8,\\,E=256$ activates about $1/32$ of them. (The attention and embedding parameters are activated for every token, so the true ratio is closer to $k/E$ only in the limit where experts dominate; we make this precise in Lesson 2.) The art of MoE design is buying a large $P_{\\text{tot}}$ — more capacity, lower loss — while holding $P_{\\text{act}}$, and therefore cost, fixed.\nQuantity Dense model MoE model Capacity / stored knowledge $P$ $P_{\\text{tot}}$ (large) Per-token compute $P$ $P_{\\text{act}}\\approx\\tfrac{k}{E}\\,P_{\\text{tot,experts}}+(\\text{attn, etc.})$ Are they equal? yes no — decoupled by the router Figure 2. Total parameters and activated parameters, pried apart.\n4. Quantifying the two phases: FLOPs and memory The dense course quantified training vs. inference with $6PD$ vs. $2P$. The MoE version of that accounting is the same arithmetic with a crucial substitution: compute scales with $P_{\\text{act}}$, memory scales with $P_{\\text{tot}}$. This split is the source of nearly every practical property of MoE models.\nCompute. Because only $k$ experts run per token, the forward pass costs\n$$ \\approx 2\\,P_{\\text{act}}\\ \\text{FLOPs/token},\\qquad\\text{training step}\\approx 6\\,P_{\\text{act}}\\ \\text{FLOPs/token}, $$and total pretraining compute is $C\\approx 6\\,P_{\\text{act}}\\,D$. The router itself adds a negligible $O(Ed)$ per token (an $E\\times d$ matrix-vector product), and the balancing machinery is cheap. So an MoE with $P_{\\text{tot}}=8\\times$ a dense model\u0026rsquo;s parameters but $P_{\\text{act}}$ equal to it costs about the same FLOPs per token to train and run — while having far more capacity. This is the entire economic case for MoE: more parameters at (nearly) constant compute.\nMemory. Here the asymmetry bites in the other direction. Every expert must be resident — stored in memory and, in training, accompanied by its gradient and optimizer moments — because the router may send some token to any expert in a given batch. So memory scales with $P_{\\text{tot}}$, not $P_{\\text{act}}$:\n$$ \\text{training memory}\\ \\approx\\ 16\\ \\text{bytes/param}\\times P_{\\text{tot}}\\ +\\ (\\text{activations}),\\qquad \\text{inference memory}\\ \\approx\\ (1\\text{–}2\\ \\text{bytes/param})\\times P_{\\text{tot}}\\ +\\ (\\text{KV cache}). $$The factor-16 figure (parameter + gradient + two Adam moments + a master copy, per the dense course\u0026rsquo;s Lesson 8) now multiplies the large number $P_{\\text{tot}}$. This is why MoE models are cheap to compute but expensive to hold — and why they are almost always trained and served across many devices, with experts sharded among them (Lesson 7), rather than on a single accelerator.\nTakeaway. Dense: one number $P$ drives both compute ($6PD$) and memory ($16P$). MoE: compute follows $P_{\\text{act}}$, memory follows $P_{\\text{tot}}$, and the gap between them — the sparsity ratio $E/k$ — is exactly what you are buying. \u0026ldquo;Cheap FLOPs, expensive bytes\u0026rdquo; is the MoE signature.\nFigure 3. Cheap FLOPs, expensive bytes.\n5. The three kinds of numbers, MoE edition The dense course sorted every quantity into parameters (learned), configuration (chosen), and state (transient). MoE adds new members to each category but breaks none of the boundaries.\n(a) Parameters $\\theta$ — learned. Everything from the dense model, plus the $E$ expert FFNs per MoE layer and the router matrix $W_g\\in\\mathbb R^{E\\times d}$ per MoE layer. The router is the only genuinely new kind of learned object in the architecture, and it is minuscule — $Ed$ parameters against the experts\u0026rsquo; $\\approx 2Ed\\,d_{ff}$ — yet it controls everything. Tagged [LEARNED].\n(b) Configuration — chosen. Everything from the dense model, plus the number of experts $E$, the number selected per token $k$, which layers are MoE vs. dense, the capacity factor $C_f$ (Lesson 6), the auxiliary-loss weight $\\alpha$ (Lesson 5), and the number of shared/always-on experts (Lesson 2). These set the shape of $\\theta$ and the rules of routing but are not entries of $\\theta$. Tagged [CONFIG].\n(c) State — transient. Everything from the dense model, plus the per-token router logits and gate weights, the top-$k$ assignment indices $\\mathcal T_k$, the dispatch/combine tensors that physically route tokens to experts (Lesson 6), and the per-batch expert-load counts used by the balancing loss. None of these are part of the deployed model; all are recomputed per batch and discarded. Tagged [STATE].\nA crisp operational test, unchanged from the dense course: if the optimizer step writes to it, it is a parameter; if a human fixes it outside the loop, it is configuration; if it exists only to make a run go and is then thrown away, it is state. The routing decision is the archetypal new piece of [STATE] — it is computed fresh for every token of every batch and never stored in the model.\n6. The mental model, restated precisely Picture the dense model as a single function evaluated everywhere. The MoE replaces one sub-layer of it with a committee whose membership is chosen per token by a learned chair (the router). At training time the whole committee is present — stored, differentiated, updated — because over a batch every member gets some work; the gradient writes to all of $\\theta$. At inference time, for any single token, only the $k$ members the chair calls on actually do anything; the rest sit idle, occupying memory but spending no compute. Training writes to the full committee; inference reads a per-token sub-committee. The two dense invariants (\u0026ldquo;training writes, inference reads\u0026rdquo;; \u0026ldquo;learned / chosen / transient\u0026rdquo;) survive intact — we have only made the forward pass route.\nConclusion A Mixture-of-Experts LLM is the same function $f(x;\\theta)$ as a dense LLM with one component changed: the per-block feed-forward network is replaced by $E$ expert FFNs plus a router that, per token, selects $k\\ll E$ of them and combines their outputs as $\\mathrm{MoE}(\\mathbf x)=\\sum_{i\\in\\mathcal T_k(\\mathbf x)}g_i(\\mathbf x)E_i(\\mathbf x)$. This conditional computation splits the single dense parameter count $P$ into two: total parameters $P_{\\text{tot}}$ (capacity, memory, $\\propto$ everything stored) and activated parameters $P_{\\text{act}}$ (per-token cost, FLOPs, $\\approx\\tfrac{k}{E}$ of the experts). Compute follows $P_{\\text{act}}$ ($\\approx 6P_{\\text{act}}D$ to train, $2P_{\\text{act}}$/token to run); memory follows $P_{\\text{tot}}$. That \u0026ldquo;cheap FLOPs, expensive bytes\u0026rdquo; asymmetry is the economic case for MoE and the root of everything that follows. The three number-categories are inherited intact, with the router as the only new kind of learned object and the routing decision as the archetypal new piece of state.\nThe next three lessons open the box. We begin at the component MoE actually replaces — the dense FFN — and grow it into a layer of experts.\n← Previous: (none — this is the first lesson) · Course index · Next: Lesson 2 — From the Dense FFN to a Layer of Experts → Appendices: A — MoE Family · B — Dense vs. MoE · C — Acronyms The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/tutorials/moe-lessons/lesson-01-the-big-picture/","summary":"\u003ch1 id=\"lesson-1--the-big-picture-sparse-vs-dense\"\u003eLesson 1 — The Big Picture: Sparse vs. Dense\u003c/h1\u003e\n\u003cp\u003e\u003cem\u003eA 10-lesson technical course on the mechanics of Mixture-of-Experts language models.\u003c/em\u003e\n\u003cem\u003eAudience: undergraduate level, already comfortable with dense decoder-only LLMs (you know next-token prediction, attention, the feed-forward network, and fine-tuning). Prerequisites: linear algebra (matrix products), multivariate calculus (gradients, the chain rule), basic probability. We use standard notation and define each new symbol as it appears.\u003c/em\u003e\u003c/p\u003e\n\u003chr\u003e\n\u003ch2 id=\"course-table-of-contents\"\u003eCourse table of contents\u003c/h2\u003e\n\u003ctable\u003e\n  \u003cthead\u003e\n      \u003ctr\u003e\n          \u003cth\u003e#\u003c/th\u003e\n          \u003cth\u003eLesson\u003c/th\u003e\n          \u003cth\u003eCore technical content\u003c/th\u003e\n      \u003c/tr\u003e\n  \u003c/thead\u003e\n  \u003ctbody\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e1\u003c/td\u003e\n          \u003ctd\u003e\u003cstrong\u003eThe Big Picture\u003c/strong\u003e \u003cem\u003e(this file)\u003c/em\u003e\u003c/td\u003e\n          \u003ctd\u003eConditional computation; $P_{\\text{tot}}$ vs. $P_{\\text{act}}$; compute $\\approx 6P_{\\text{act}}D$ but memory $\\propto P_{\\text{tot}}$; the dense contrast\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e2\u003c/td\u003e\n          \u003ctd\u003e\u003ca href=\"/tutorials/moe-lessons/lesson-02-from-ffn-to-experts/\"\u003eFrom the Dense FFN to a Layer of Experts\u003c/a\u003e\n\u003c/td\u003e\n          \u003ctd\u003eThe FFN holds $\\tfrac23$ of params; $E$ replicated experts; the MoE-layer map; placement; shared / fine-grained experts\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e3\u003c/td\u003e\n          \u003ctd\u003e\u003ca href=\"/tutorials/moe-lessons/lesson-03-the-router-and-gating/\"\u003eThe Router / Gating Network\u003c/a\u003e\n\u003c/td\u003e\n          \u003ctd\u003e$g(\\mathbf x)=\\mathrm{softmax}(W_g\\mathbf x)$; top-$k$; noisy top-$k$; softmax-before vs. -after; the only new learned mechanism\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e4\u003c/td\u003e\n          \u003ctd\u003e\u003ca href=\"/tutorials/moe-lessons/lesson-04-sparse-computation-and-combine/\"\u003eSparse Top-k Computation \u0026amp; the Combine\u003c/a\u003e\n\u003c/td\u003e\n          \u003ctd\u003eThe weighted sum $\\sum_{i\\in\\mathcal T_k}g_i E_i(\\mathbf x)$; sparse compute, dense memory; gradient through a discrete choice\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e5\u003c/td\u003e\n          \u003ctd\u003e\u003ca href=\"/tutorials/moe-lessons/lesson-05-load-balancing-and-auxiliary-losses/\"\u003eLoad Balancing \u0026amp; Auxiliary Losses\u003c/a\u003e\n\u003c/td\u003e\n          \u003ctd\u003eExpert collapse; importance/load loss; Switch loss $\\alpha E\\sum_i f_iP_i$; z-loss; aux-loss-free bias\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e6\u003c/td\u003e\n          \u003ctd\u003e\u003ca href=\"/tutorials/moe-lessons/lesson-06-capacity-and-token-dropping/\"\u003eCapacity \u0026amp; Token Dropping\u003c/a\u003e\n\u003c/td\u003e\n          \u003ctd\u003eCapacity factor $C_f$; capacity $=C_fkT/E$; overflow/dropping; dispatch \u0026amp; combine tensors\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e7\u003c/td\u003e\n          \u003ctd\u003e\u003ca href=\"/tutorials/moe-lessons/lesson-07-expert-parallelism/\"\u003eExpert Parallelism\u003c/a\u003e\n\u003c/td\u003e\n          \u003ctd\u003eSharding experts; the two \u003cstrong\u003eall-to-all\u003c/strong\u003e collectives; communication-bound training; GShard\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e8\u003c/td\u003e\n          \u003ctd\u003e\u003ca href=\"/tutorials/moe-lessons/lesson-08-training-dynamics-and-scaling/\"\u003eTraining Dynamics \u0026amp; Scaling\u003c/a\u003e\n\u003c/td\u003e\n          \u003ctd\u003eRouter instability \u0026amp; z-loss; router precision; emergent specialization; params-at-fixed-FLOPs scaling\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e9\u003c/td\u003e\n          \u003ctd\u003e\u003ca href=\"/tutorials/moe-lessons/lesson-09-finetuning-moe/\"\u003eFine-Tuning MoE\u003c/a\u003e\n\u003c/td\u003e\n          \u003ctd\u003eOverfitting of sparse models; expert dropout; selective FT; router freezing; vs. dense SFT/RLHF\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e10\u003c/td\u003e\n          \u003ctd\u003e\u003ca href=\"/tutorials/moe-lessons/lesson-10-inference-in-depth/\"\u003eInference in Depth\u003c/a\u003e\n\u003c/td\u003e\n          \u003ctd\u003eMemory-vs-compute asymmetry; batch routing \u0026amp; load imbalance; unchanged KV cache; offloading \u0026amp; serving\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eA\u003c/td\u003e\n          \u003ctd\u003e\u003ca href=\"/tutorials/moe-lessons/appendix-a-moe-family/\"\u003eAppendix A — MoE Family\u003c/a\u003e\n\u003c/td\u003e\n          \u003ctd\u003eShazeer, GShard, Switch, GLaM, Mixtral, DeepSeek-MoE along the axes that differ\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eB\u003c/td\u003e\n          \u003ctd\u003e\u003ca href=\"/tutorials/moe-lessons/appendix-b-dense-vs-moe/\"\u003eAppendix B — Dense vs. MoE\u003c/a\u003e\n\u003c/td\u003e\n          \u003ctd\u003eEvery contrast in one decision-oriented table\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eC\u003c/td\u003e\n          \u003ctd\u003e\u003ca href=\"/tutorials/moe-lessons/appendix-c-acronyms/\"\u003eAppendix C — Acronyms\u003c/a\u003e\n\u003c/td\u003e\n          \u003ctd\u003eEvery acronym and symbol, expanded\u003c/td\u003e\n      \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp\u003eTwo invariants run through every lesson and are worth stating before anything else:\u003c/p\u003e","title":"The Big Picture"},{"content":"Lesson 1 — The Big Picture: Encoder vs. Decoder A 10-lesson technical course on the mechanics of encoder language models. Audience: comfortable with decoder-style LLMs at a high level; new to the specifics of encoders. Prerequisites: linear algebra (matrix products), multivariate calculus (gradients, the chain rule), and basic probability (distributions, expectation). We use standard notation throughout and define it as we go.\nCourse table of contents # Lesson Core technical content 1 The Big Picture (this file) The object $g(x;\\theta):\\mathcal V^{n}\\to\\mathbb R^{n\\times d}$; bidirectional context; why there is no autoregressive factorization; the running decoder contrast 2 Tokenization \u0026amp; the Encoder Input WordPiece; [CLS]/[SEP]/[MASK]; token + segment + position embeddings; learned-absolute positions 3 Bidirectional Self-Attention $\\mathrm{softmax}(QK^\\top/\\sqrt{d_k})V$ with no causal mask; the full $n\\times n$ matrix; no KV cache; padding masks 4 The Encoder Block \u0026amp; Stack Post-LN vs. pre-LN; the residual stream; contextual embeddings; how [CLS] summarizes a sequence 5 Masked Language Modeling The MLM loss; 80/10/10 corruption; why bidirectionality demands denoising; NSP; masked cross-entropy 6 From Representations to Predictions MLM head + weight tying; [CLS]/mean pooling; token- vs. sequence-level heads 7 Evaluating an Encoder Masked cross-entropy; why perplexity fails; pseudo-perplexity; GLUE/SuperGLUE 8 Pretraining Dynamics AdamW engine; the 15% signal-density problem; sequence packing; the RoBERTa recipe 9 Fine-Tuning Downstream Classification, NER, span QA, sentence pairs; task heads; full FT vs. adapters 10 Practice \u0026amp; Landscape Sentence-BERT; retrieval/RAG; RoBERTa/DeBERTa/ELECTRA/ModernBERT; encoder–decoder; which to choose A Appendix A — Encoder Family BERT → ModernBERT, organized by what each model changed B Appendix B — Encoder vs. Decoder Cheat Sheet Every contrast in the course, in one decision table C Appendix C — Acronyms Every acronym and symbol, expanded in plain language Two invariants run through every lesson and are worth stating before anything else:\nAn encoder reads bidirectionally and emits representations, not a next-token distribution. Its output is a matrix $H\\in\\mathbb R^{n\\times d}$ of context-aware vectors, one per input token, not a probability over the vocabulary. The pretraining objective must be a denoising task. Because every token can see every other, \u0026ldquo;predict the next token\u0026rdquo; is trivially solved by looking ahead. The model is instead trained to reconstruct corrupted input — this single fact reshapes the objective, the output head, the evaluation, and the fine-tuning story. This course assumes you have met the decoder. If \u0026ldquo;autoregressive,\u0026rdquo; \u0026ldquo;KV cache,\u0026rdquo; or \u0026ldquo;cross-entropy next-token loss\u0026rdquo; are unfamiliar, read the companion decoder course first; here we treat the decoder as the known reference point and measure the encoder against it.\n1. Two ways to be a language model A decoder-only LLM is a function that reads the text so far and outputs a probability for every token in the vocabulary as the next token. Formally,\n$$ f:\\ \\mathcal V^{\\le n}\\times\\Theta\\to\\Delta^{V-1},\\qquad \\hat{\\mathbf p}=f(x;\\theta), $$mapping a sequence to a point on the probability simplex $\\Delta^{V-1}$ — a categorical distribution over the next token. Its defining property is the autoregressive factorization, the chain rule of probability:\n$$ P_\\theta(x_1,\\dots,x_n)=\\prod_{t=1}^{n}P_\\theta(x_t\\mid x_{\\lt t}),\\qquad x_{\\lt t}:=(x_1,\\dots,x_{t-1}). $$Each factor conditions only on the past. That is not an aesthetic choice; it is what makes the factorization an exact identity, and it forces every token to be blind to its future.\nAn encoder drops that constraint entirely. Its job is not to predict what comes next but to represent what is here. It is a function\n$$ \\boxed{\\ g:\\ \\mathcal V^{n}\\times\\Theta\\to\\mathbb R^{n\\times d},\\qquad H=g(x;\\theta),\\ } $$mapping a length-$n$ token sequence to a length-$n$ sequence of $d$-dimensional vectors. Row $i$ of the output, $H_{i,:}\\in\\mathbb R^{d}$, is the contextual representation of token $i$: a vector that encodes what token $i$ means in this particular context, having looked at all the other tokens — both before and after it. The same word produces different output rows in different sentences (\u0026ldquo;bank\u0026rdquo; by a river vs. \u0026ldquo;bank\u0026rdquo; with money), which is the entire point.\nNotice what is not in the output type. There is no $\\Delta^{V-1}$, no distribution over the vocabulary, no \u0026ldquo;next token.\u0026rdquo; An encoder, on its own, predicts nothing. It produces a richer numerical description of the input that downstream heads (Lesson 6) then turn into predictions — a class label, a tag per token, an answer span, a similarity score. This is the sense in which decoders generate and encoders understand.\nInvariant. A decoder\u0026rsquo;s native output is a distribution over the next token; an encoder\u0026rsquo;s native output is a matrix of contextual vectors. Everything else in this course follows from that difference.\nWhy \u0026ldquo;bidirectional\u0026rdquo; is the crux The decoder\u0026rsquo;s causal constraint — position $t$ sees only $\\le t$ — exists so that the training-time prediction at position $t$ cannot cheat by reading the answer ($x_t$ and beyond). An encoder has no such prediction to protect, so it lifts the constraint: position $i$ attends to all positions $1,\\dots,n$. We will derive this precisely in Lesson 3, but the consequence is immediate and worth internalizing now: a representation that has seen the whole sentence is strictly more informative for understanding tasks than one that has seen only a prefix. The word \u0026ldquo;it\u0026rdquo; can be resolved by a noun that appears later; a sentiment can hinge on a final clause. Bidirectionality is what makes encoders the natural tool for classification, tagging, retrieval, and extraction.\nFigure 1. Two output types: a distribution, or a matrix.\n2. The phases, and how they differ from the decoder\u0026rsquo;s The decoder course\u0026rsquo;s central split — training writes $\\theta$, inference reads $\\theta$ — still holds. An encoder is trained by the same forward/backward/update loop, and at deployment its parameters are frozen. What changes is what each phase computes.\nTraining Given a corpus $\\mathcal D$, pretraining solves an empirical risk minimization\n$$ \\theta^\\star=\\arg\\min_{\\theta\\in\\mathbb R^{P}}\\ \\mathcal L(\\theta), $$but $\\mathcal L$ is not the next-token loss. As §3 previews and Lesson 5 derives, it is a masked reconstruction loss: corrupt a fraction of the input, run the encoder, and score how well the model recovers the original tokens at the corrupted positions. The four-step loop (forward → backward → optimizer step → apply) is identical to the decoder\u0026rsquo;s; only the loss\u0026rsquo;s source differs.\nInference Here the contrast is stark. A decoder at inference runs a loop: evaluate $f$ at the last position, sample a token, append it, evaluate again — $n$ forward passes to produce $n$ tokens, with a KV cache to avoid recomputation. An encoder at inference runs once: a single forward pass over the whole input produces all $n$ representations simultaneously. There is no token-by-token loop, no sampling, no KV cache, no temperature. You hand it a sequence; it hands you back a matrix.\nContrast with the decoder. Decoder inference is sequential and autoregressive ($O(n)$ forward passes, growing KV cache). Encoder inference is a single parallel pass ($1$ forward pass, no cache). This is why encoders are cheap to run as feature extractors and why they are useless as text generators.\nFigure 2. One pass, or n: the inference contrast.\n3. The objective, previewed The deepest difference is the training objective, so it is worth stating the logic now even though Lesson 5 does the math. A decoder is trained on next-token prediction precisely because it is causal: hiding the future is what makes the task non-trivial. Remove causality and that task collapses — if position $t$ can attend to $x_t$ itself, predicting $x_t$ is free. So a bidirectional model needs a different game.\nThe answer, introduced by BERT, is masked language modeling (MLM): replace a random ~15% of input tokens with a special [MASK] symbol (and some other corruptions, Lesson 5), run the full bidirectional encoder, and at each masked position predict the original token from its surrounding context. The loss is cross-entropy, but it is computed only at the masked positions:\n$$ \\mathcal L_{\\text{MLM}}(\\theta)=-\\,\\mathbb E_{x\\sim\\mathcal D}\\ \\sum_{i\\in\\mathcal M}\\log P_\\theta\\!\\big(x_i\\mid x_{\\setminus\\mathcal M}\\big), $$where $\\mathcal M$ is the set of masked positions and $x_{\\setminus\\mathcal M}$ is the sequence with those positions corrupted. Read it against the decoder\u0026rsquo;s loss: the decoder conditions on the past $x_{\\lt t}$ and sums over all positions; the encoder conditions on the whole corrupted context and sums over only the masked positions. That contrast — what you condition on, and which positions contribute loss — is the spine of Lessons 5 through 7.\nflowchart LR subgraph DEC[\"Decoder (autoregressive)\"] d1[\"x₁ x₂ x₃ x₄\"] --\u003e d2[\"predict x₅ from x₁..x₄(causal: past only)\"] end subgraph ENC[\"Encoder (masked denoising)\"] e1[\"x₁ [MASK] x₃ x₄\"] --\u003e e2[\"predict x₂ from ALL ofx₁,x₃,x₄ (bidirectional)\"] end 4. The three kinds of numbers (unchanged) The number-categories from the decoder course carry over without modification, and we tag them identically throughout.\n(a) Parameters $\\theta$ — learned. Every weight matrix: the token embedding $E$, the new segment and position embeddings (Lesson 2), the attention projections $W_Q,W_K,W_V,W_O$, the MLP weights, the LayerNorm gains/shifts, and the MLM output projection. Initialized randomly, updated each step. Tagged [LEARNED].\n(b) Configuration — chosen. Architectural: depth $N$, width $d$, heads $h$, MLP width $d_{ff}$, vocab $V$, max sequence length $n$. Process: learning rate, batch size, weight decay, and — uniquely for encoders — the masking rate (≈15%) and the masking recipe. Tagged [CONFIG].\n(c) State — transient. Adam moments, cached activations, the attention matrices $A\\in\\mathbb R^{n\\times n}$, and the logits at masked positions. Note what is absent relative to the decoder: there is no KV cache, because there is no autoregressive loop to cache for. Tagged [STATE].\nProperty Parameters $\\theta$ Configuration State Set by gradient updates humans / search computed during a run Encoder-specific members segment embeddings, MLM head masking rate, masking recipe attention matrix $A$; no KV cache Persists into deployed model? yes yes no Figure 3. The same three categories, with encoder-specific members.\n5. What an encoder is for It helps to fix the use-cases now, because they motivate every design choice that follows. Decoders are deployed to produce text: chat, completion, code generation, agents. Encoders are deployed to score, classify, tag, and embed:\nSequence classification — sentiment, spam, topic, intent (one label per input). Token classification — named-entity recognition, part-of-speech tagging (one label per token). Span extraction — extractive question answering (predict start/end positions in a passage). Sentence-pair tasks — entailment, paraphrase, semantic similarity. Embeddings \u0026amp; retrieval — turn a document or query into a single vector for search, clustering, or RAG. Every one of these wants a rich, whole-context representation of given text, and none of them needs to generate new text. That is exactly the encoder\u0026rsquo;s contract.\nConclusion An encoder language model is a function $g(x;\\theta):\\mathcal V^{n}\\to\\mathbb R^{n\\times d}$ that maps a token sequence to a sequence of contextual representations, each informed by the entire input bidirectionally. It differs from a decoder at every level, and all the differences descend from one root: an encoder reads rather than writes. Because it is bidirectional it has no autoregressive factorization, so it cannot be trained on next-token prediction and is instead trained by masked reconstruction; because it emits representations rather than a next-token distribution it needs task heads to make predictions; because it makes no token-by-token loop it has no KV cache and runs in a single parallel forward pass; and because it does not generate, it is evaluated and fine-tuned differently. The same number-categories — [LEARNED], [CONFIG], [STATE] — and the same training engine apply throughout; what changes is the object, the objective, and the output.\nThe next lessons open the box from the bottom up, exactly as the decoder course did, but at each layer we will hold the decoder version up beside the encoder version. Lesson 2 starts at the boundary where text becomes numbers — and finds that an encoder\u0026rsquo;s input is built differently from a decoder\u0026rsquo;s from the very first token.\n← Previous: (none — this is the first lesson) · Course index · Next: Lesson 2 — Tokenization \u0026amp; the Encoder Input → Appendices: A — Encoder Family · B — Encoder vs. Decoder Cheat Sheet · C — Acronyms The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/tutorials/encoder-deep-dive/lesson-01-the-big-picture/","summary":"\u003ch1 id=\"lesson-1--the-big-picture-encoder-vs-decoder\"\u003eLesson 1 — The Big Picture: Encoder vs. Decoder\u003c/h1\u003e\n\u003cp\u003e\u003cem\u003eA 10-lesson technical course on the mechanics of encoder language models.\u003c/em\u003e\n\u003cem\u003eAudience: comfortable with decoder-style LLMs at a high level; new to the specifics of encoders. Prerequisites: linear algebra (matrix products), multivariate calculus (gradients, the chain rule), and basic probability (distributions, expectation). We use standard notation throughout and define it as we go.\u003c/em\u003e\u003c/p\u003e\n\u003chr\u003e\n\u003ch2 id=\"course-table-of-contents\"\u003eCourse table of contents\u003c/h2\u003e\n\u003ctable\u003e\n  \u003cthead\u003e\n      \u003ctr\u003e\n          \u003cth\u003e#\u003c/th\u003e\n          \u003cth\u003eLesson\u003c/th\u003e\n          \u003cth\u003eCore technical content\u003c/th\u003e\n      \u003c/tr\u003e\n  \u003c/thead\u003e\n  \u003ctbody\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e1\u003c/td\u003e\n          \u003ctd\u003e\u003cstrong\u003eThe Big Picture\u003c/strong\u003e \u003cem\u003e(this file)\u003c/em\u003e\u003c/td\u003e\n          \u003ctd\u003eThe object $g(x;\\theta):\\mathcal V^{n}\\to\\mathbb R^{n\\times d}$; bidirectional context; why there is no autoregressive factorization; the running decoder contrast\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e2\u003c/td\u003e\n          \u003ctd\u003e\u003ca href=\"/tutorials/encoder-deep-dive/lesson-02-tokenization-and-embeddings/\"\u003eTokenization \u0026amp; the Encoder Input\u003c/a\u003e\n\u003c/td\u003e\n          \u003ctd\u003eWordPiece; \u003ccode\u003e[CLS]\u003c/code\u003e/\u003ccode\u003e[SEP]\u003c/code\u003e/\u003ccode\u003e[MASK]\u003c/code\u003e; token + segment + position embeddings; learned-absolute positions\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e3\u003c/td\u003e\n          \u003ctd\u003e\u003ca href=\"/tutorials/encoder-deep-dive/lesson-03-bidirectional-self-attention/\"\u003eBidirectional Self-Attention\u003c/a\u003e\n\u003c/td\u003e\n          \u003ctd\u003e$\\mathrm{softmax}(QK^\\top/\\sqrt{d_k})V$ with \u003cstrong\u003eno causal mask\u003c/strong\u003e; the full $n\\times n$ matrix; no KV cache; padding masks\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e4\u003c/td\u003e\n          \u003ctd\u003e\u003ca href=\"/tutorials/encoder-deep-dive/lesson-04-the-encoder-block-and-stack/\"\u003eThe Encoder Block \u0026amp; Stack\u003c/a\u003e\n\u003c/td\u003e\n          \u003ctd\u003ePost-LN vs. pre-LN; the residual stream; contextual embeddings; how \u003ccode\u003e[CLS]\u003c/code\u003e summarizes a sequence\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e5\u003c/td\u003e\n          \u003ctd\u003e\u003ca href=\"/tutorials/encoder-deep-dive/lesson-05-masked-language-modeling/\"\u003eMasked Language Modeling\u003c/a\u003e\n\u003c/td\u003e\n          \u003ctd\u003eThe MLM loss; 80/10/10 corruption; why bidirectionality demands denoising; NSP; masked cross-entropy\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e6\u003c/td\u003e\n          \u003ctd\u003e\u003ca href=\"/tutorials/encoder-deep-dive/lesson-06-representations-to-predictions/\"\u003eFrom Representations to Predictions\u003c/a\u003e\n\u003c/td\u003e\n          \u003ctd\u003eMLM head + weight tying; \u003ccode\u003e[CLS]\u003c/code\u003e/mean pooling; token- vs. sequence-level heads\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e7\u003c/td\u003e\n          \u003ctd\u003e\u003ca href=\"/tutorials/encoder-deep-dive/lesson-07-loss-and-evaluation/\"\u003eEvaluating an Encoder\u003c/a\u003e\n\u003c/td\u003e\n          \u003ctd\u003eMasked cross-entropy; why perplexity fails; pseudo-perplexity; GLUE/SuperGLUE\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e8\u003c/td\u003e\n          \u003ctd\u003e\u003ca href=\"/tutorials/encoder-deep-dive/lesson-08-pretraining-dynamics/\"\u003ePretraining Dynamics\u003c/a\u003e\n\u003c/td\u003e\n          \u003ctd\u003eAdamW engine; the 15% signal-density problem; sequence packing; the RoBERTa recipe\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e9\u003c/td\u003e\n          \u003ctd\u003e\u003ca href=\"/tutorials/encoder-deep-dive/lesson-09-finetuning-downstream/\"\u003eFine-Tuning Downstream\u003c/a\u003e\n\u003c/td\u003e\n          \u003ctd\u003eClassification, NER, span QA, sentence pairs; task heads; full FT vs. adapters\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e10\u003c/td\u003e\n          \u003ctd\u003e\u003ca href=\"/tutorials/encoder-deep-dive/lesson-10-practice-and-landscape/\"\u003ePractice \u0026amp; Landscape\u003c/a\u003e\n\u003c/td\u003e\n          \u003ctd\u003eSentence-BERT; retrieval/RAG; RoBERTa/DeBERTa/ELECTRA/ModernBERT; encoder–decoder; which to choose\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eA\u003c/td\u003e\n          \u003ctd\u003e\u003ca href=\"/tutorials/encoder-deep-dive/appendix-a-encoder-family/\"\u003eAppendix A — Encoder Family\u003c/a\u003e\n\u003c/td\u003e\n          \u003ctd\u003eBERT → ModernBERT, organized by what each model changed\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eB\u003c/td\u003e\n          \u003ctd\u003e\u003ca href=\"/tutorials/encoder-deep-dive/appendix-b-encoder-vs-decoder/\"\u003eAppendix B — Encoder vs. Decoder Cheat Sheet\u003c/a\u003e\n\u003c/td\u003e\n          \u003ctd\u003eEvery contrast in the course, in one decision table\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eC\u003c/td\u003e\n          \u003ctd\u003e\u003ca href=\"/tutorials/encoder-deep-dive/appendix-c-acronyms/\"\u003eAppendix C — Acronyms\u003c/a\u003e\n\u003c/td\u003e\n          \u003ctd\u003eEvery acronym and symbol, expanded in plain language\u003c/td\u003e\n      \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp\u003eTwo invariants run through every lesson and are worth stating before anything else:\u003c/p\u003e","title":"The Big Picture: Encoder vs. Decoder"},{"content":"Chapter 1 — What These Machines Actually Are Before we open the hood on any particular model, it helps to settle what kind of thing we are looking at. The systems in this book go by intimidating names and produce uncanny results, but underneath they are far simpler in concept than they appear. This chapter lays down the three ideas that the rest of the book will lean on constantly. None of them requires any mathematics. All of them are easy to misunderstand, which is precisely why we start here.\nA model is just a function Strip away the branding and an AI model is a function: something that takes an input and produces an output. You put text in; you get text out. You put a prompt in; you get an image out. That is the whole shape of it. What makes these particular functions remarkable is not the idea of input-and-output — your pocket calculator does that — but the sheer richness of what they have absorbed about the world in order to produce sensible outputs.\nThe richness lives in what we will call the model\u0026rsquo;s settings: an enormous collection of internal numbers, often numbering in the billions, that determine exactly how an input gets transformed into an output. People call these numbers parameters. You can think of them as billions of tiny knobs. Turn the knobs one way and the function behaves one way; turn them another way and it behaves differently. A \u0026ldquo;trained model\u0026rdquo; is nothing more than one very particular setting of all those knobs — a setting that happens to produce useful, coherent results.\nThis already tells us something important. The model has no secret database it consults, no little library it flips through. Everything it \u0026ldquo;knows\u0026rdquo; is baked into the positions of those knobs. When a language model tells you the capital of France, it is not looking it up. The fact that Paris is the answer has been pressed into the arrangement of its knobs, the same way a song is pressed into the grooves of a record.\nThe most important distinction in the book: training versus using Here is the single idea that clears up more confusion than any other. There are two completely separate phases in the life of one of these models, and they could not be more different.\nThe first phase is training. This is how the model is built. Over weeks or months, on vast banks of specialized computers, the model is shown staggering quantities of material — for a language model, trillions of words; for an image model, billions of pictures — and its billions of knobs are slowly, patiently adjusted until its outputs become good. Training is the only time the model changes. It is expensive, it is slow, and it happens once.\nThe second phase is using the model — the technical word is inference. This is what happens when you type a question and get an answer. And here is the crucial part: during this phase, the model does not change at all. Its knobs are frozen solid. It reads your input, runs it through the fixed function, and produces an output. Then it does it again for the next input. Nothing about the model itself is altered, no matter how long or how cleverly you talk to it.\nThere is a phrase worth memorizing: training writes; using reads. Building the model writes the knobs into place. Using the model only reads them.\nThis distinction explains a question almost everyone asks: does the model learn from our conversation? The answer is no. The reason a chatbot seems to remember what you said three paragraphs ago is not that it learned anything — it is simply that your earlier words are still sitting right there in front of it as part of the input, the way a person re-reading the top of a page is not someone whose brain rewired itself mid-sentence. When the conversation ends and a new one begins, that context is gone, and the model is exactly what it was before. It never learned you; it only ever re-read the page.\nThis also explains the wild difference in cost between the two phases. Building one of these models can cost a fortune and consume the output of a small power plant. Running it, by comparison, is cheap — cheap enough that a company can train a model once and then serve it to millions of people. The hard, costly work is front-loaded into a single act of construction.\nFigure 1. Training writes; using reads.\nThree kinds of numbers, never to be confused Inside any of these systems there are numbers everywhere, and they are not all the same kind of thing. Sorting them into three buckets is one of the most clarifying habits you can develop, and the original courses tag every number this way for exactly that reason.\nThe first bucket is the learned numbers — the knobs, the parameters. These are adjusted by training and by nothing else. They are the model. When someone says \u0026ldquo;a model with seventy billion parameters,\u0026rdquo; they are counting the knobs in this bucket.\nThe second bucket is the chosen numbers — the settings a human picks before or during training, and which the model never adjusts on its own. How many knobs the model should have, how big to make it, how adventurous to let it be when it answers — these are human decisions. They shape the model and the process, but they are not learned; they are dialed in.\nThe third bucket is the scratch numbers — temporary working values that exist only during a single run and are thrown away the instant it finishes. When the model processes your sentence, it generates a great deal of intermediate arithmetic to get from input to output. None of it is kept. It is the mental scratch paper the model covers in pencil and then discards.\nKeep these three apart and a surprising amount of mystery evaporates. \u0026ldquo;Did the model learn that?\u0026rdquo; becomes a precise question: did it change a knob in the first bucket (no, not while you were talking to it), or are you seeing the effect of a human setting in the second bucket, or just scratch work in the third? Throughout this book, when something seems puzzling, ask which bucket of numbers is really involved.\nFigure 2. Three kinds of numbers, and how to tell them apart.\nWhy \u0026ldquo;guess the next thing\u0026rdquo; turns out to be enough There is one more foundational idea, and it is the one that makes everything else possible. Most of the models in this book are trained on a goal that sounds almost insultingly modest: predict what comes next.\nFor a language model, \u0026ldquo;what comes next\u0026rdquo; is the next word. You show it a stretch of text with the next word hidden, and you train it to guess that word. That is the entire training game, repeated across an unfathomable amount of text. It seems impossible that such a narrow exercise could give rise to essays, working code, translation, and argument. The resolution to that paradox is the intellectual heart of the whole field, so it is worth stating carefully.\nTo guess the next word well, across every imaginable sentence and topic, a model is quietly forced to learn far more than word-frequency trivia. To finish \u0026ldquo;the boiling point of water at sea level is one hundred degrees ___\u0026rdquo; it has to have absorbed some physics. To finish \u0026ldquo;she opened the door and gasped, because standing there was ___\u0026rdquo; it has to track narrative and surprise. To finish a half-written function it has to grasp the logic of code. Predicting the next word turns out to be a back door into modeling the whole structure of language — and, through language, a great deal of the structure of the world. The modest goal is a Trojan horse.\nThis is why you will see the phrase \u0026ldquo;next-token prediction\u0026rdquo; treated with such reverence. It is not a party trick bolted onto something cleverer. It is the cleverness. (Image models, as we will see in Chapter 8, are trained on a different but equally humble goal — \u0026ldquo;guess the noise\u0026rdquo; — with the same surprising payoff.)\nFigure 3. What guessing the next word quietly forces a model to learn.\nWhat to carry forward We now have the frame. A model is a function whose behavior is set by billions of learned knobs. It is built once, in an expensive training phase that writes those knobs, and then used in a frozen state that only reads them — which is why it does not learn from your conversation. Its numbers come in three kinds: learned, chosen, and scratch. And the engine that drives most of it is the unreasonably powerful goal of predicting what comes next.\nWith that settled, we can start opening the box. The first thing we need to understand is how something as fuzzy and human as language gets turned into something a machine can compute with at all. That is the subject of Chapter 2 .\nContents · Chapter 2 · From Words to Numbers → The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/tutorials/inside-the-machines/chapter-01-what-these-machines-are/","summary":"\u003ch1 id=\"chapter-1--what-these-machines-actually-are\"\u003eChapter 1 — What These Machines Actually Are\u003c/h1\u003e\n\u003cp\u003eBefore we open the hood on any particular model, it helps to settle what kind of thing we are looking at. The systems in this book go by intimidating names and produce uncanny results, but underneath they are far simpler in concept than they appear. This chapter lays down the three ideas that the rest of the book will lean on constantly. None of them requires any mathematics. All of them are easy to misunderstand, which is precisely why we start here.\u003c/p\u003e","title":"What These Machines Actually Are"},{"content":"Lesson 1 — Why Two Stacks? The Sequence-to-Sequence Problem A 10-lesson technical course on encoder–decoder (seq2seq) transformers. Audience: you already know the decoder-only transformer — self-attention, next-token prediction, softmax cross-entropy — and want the encoder–decoder nuances. Prerequisites: linear algebra, multivariate calculus, basic probability. We reuse standard notation and define what is new as we go.\nCourse table of contents # Lesson Core technical content 1 Why Two Stacks? (this file) The conditional map $P_\\theta(y\\mid x)$; encoder-only vs. decoder-only vs. encoder–decoder; why seq2seq wants two stacks 2 The Encoder Stack Bidirectional self-attention (no mask); $H_{\\text{enc}}\\in\\mathbb R^{n\\times d}$ as fixed memory; padding masks; vs. BERT 3 The Decoder Stack Causal self-attention revisited; the three-sublayer block; what is added vs. a decoder-only model 4 Cross-Attention $Q=H_{\\text{dec}}W_Q$, $K,V=H_{\\text{enc}}W_{K},H_{\\text{enc}}W_V$; the non-square $m\\times n$ map; gradient into the encoder 5 The Encoder–Decoder Block Self-attn + cross-attn + FFN; full architecture; parameter budget; T5 vs. BART 6 Prefix-LM The prefix mask; folding two stacks into one; the encoder-only ↔ decoder-only spectrum 7 Pretraining Objectives T5 span corruption with sentinels; BART\u0026rsquo;s five noises; vs. MLM and next-token CE 8 Training: Teacher Forcing Shifted decoder inputs; CE over target positions; parallel training; gradient through cross-attention; exposure bias 9 Inference Encode-once/decode-many; the two KV caches; beam search; coverage and copy mechanisms 10 Fine-Tuning \u0026amp; Distillation Text-to-text framing; translation/summarization; BLEU/ROUGE; distilling T5-small A Appendix A — Formulas All key equations, collected by lesson B Appendix B — Architecture Comparison The three families along masking, conditioning, objective, use-case axes C Appendix C — Acronyms Every acronym and symbol, expanded One invariant runs through every lesson and is worth stating before anything else:\nThe encoder compresses context; the decoder generates conditioned on it; cross-attention is the only new mechanism.\nTwo more are inherited from the decoder-only story and reused throughout: training writes, inference reads, and parameters are learned, configuration is chosen, state is transient (tagged [LEARNED]/[CONFIG]/[STATE]).\n1. Two sequences, not one The decoder-only model you already know is a function of one growing sequence: it reads $x_{\\lt t}$ and predicts $x_t$, scoring a whole document by the chain rule $P_\\theta(x_{1:n})=\\prod_t P_\\theta(x_t\\mid x_{\\lt t})$. Many real tasks, though, are not \u0026ldquo;continue this text\u0026rdquo; — they are \u0026ldquo;given this sequence, produce that one.\u0026rdquo; Translate an English sentence into German. Summarize a 2,000-word article into three sentences. Answer a question given a passage. These are sequence-to-sequence (seq2seq) tasks: an input sequence $x=(x_1,\\dots,x_n)$ over one vocabulary maps to an output sequence $y=(y_1,\\dots,y_m)$ over another (or the same), and crucially $n\\neq m$ in general.\nThe object we want to model is therefore a conditional distribution. Factorizing the output autoregressively, but conditioning every factor on the entire input:\n$$ P_\\theta(y\\mid x)=\\prod_{t=1}^{m} P_\\theta\\big(y_t\\mid y_{\\lt t},\\,x\\big),\\qquad y_{\\lt t}:=(y_1,\\dots,y_{t-1}). $$Compare this with the decoder-only factorization. Two things changed. First, the product runs over the target positions $1\\dots m$, not the input. Second — and this is the whole game — every factor is conditioned on $x$ in full, not just on the part of it seen so far. The input is known up front and never grows; the output is generated left to right. That asymmetry between a fully-available input and an incrementally-produced output is the structural fact the architecture has to respect, and it is why one uniform stack is an awkward fit.\n2. Three ways to wire a transformer There are three canonical ways to point the same transformer machinery at a problem. They differ only in who may attend to whom (the mask) and what is conditioned on what; everything else — embeddings, multi-head attention, FFNs, residuals, softmax — is shared. Keeping these three straight is the prerequisite for everything that follows.\nEncoder-only (e.g. BERT). A single stack with bidirectional self-attention: every token sees every other token, past and future. It maps a sequence to a sequence of contextual vectors $H\\in\\mathbb R^{n\\times d}$, used for classification, tagging, or retrieval. It is an excellent reader — it builds rich representations of a known input — but it has no built-in way to generate a variable-length output one token at a time. There is no autoregressive factorization, because nothing is masked.\nDecoder-only (e.g. GPT). A single stack with causal self-attention: token $t$ sees only $\\le t$. This is a superb generator — the causal mask makes the autoregressive factorization valid and lets all positions train in parallel. To do a seq2seq task with it, you concatenate: feed [x ; separator ; y] as one stream and let the model continue. Conditioning on $x$ is real but implicit — it happens through the same causal self-attention that also generates, and the input is read strictly left-to-right like everything else.\nEncoder–decoder (e.g. T5, BART, the original Transformer). Two stacks. A bidirectional encoder reads $x$ once and compresses it into a memory $H_{\\text{enc}}\\in\\mathbb R^{n\\times d}$. A causal decoder generates $y$ left to right, and at every layer reaches into that memory through a new sublayer — cross-attention — whose queries come from the decoder and whose keys and values come from the encoder. Conditioning on $x$ is explicit and separated from generation.\nflowchart LR subgraph EO[\"Encoder-only (BERT)\"] x1[\"x (bidirectional)\"] --\u003e h1[\"H — representations\"] end subgraph DO[\"Decoder-only (GPT)\"] c1[\"[x ; sep ; y] (causal)\"] --\u003e g1[\"next-token over the stream\"] end subgraph ED[\"Encoder–decoder (T5/BART)\"] ex[\"x → Encoder(bidirectional)\"] --\u003e mem[\"H_enc memory\"] mem -.cross-attention.-\u003e dec[\"Decoder(causal) → y\"] end Figure 1. Three ways to point the same machinery at a problem.\n3. Why two stacks, precisely Why not just use the decoder-only concatenation trick, which clearly works (and is what most chat LLMs do)? The encoder–decoder split buys three concrete things, each a direct consequence of treating input and output differently.\n(a) Bidirectional encoding of the input. When the entire input is available before generation begins — which is exactly the seq2seq setting — there is no reason to read it left-to-right. A token in the source sentence can be disambiguated by words that come after it. The encoder\u0026rsquo;s unmasked self-attention lets every input token attend to every other, producing a representation that a causal stream simply cannot match for the input portion. The decoder-only model pays for its uniformity by reading the input causally too.\n(b) A reusable, compute-once memory. The encoder runs once per input and produces $H_{\\text{enc}}$, which is then frozen for the entire generation. Every one of the $m$ decoding steps attends against the same $H_{\\text{enc}}$. We will see in Lesson 9 that this makes the encoder\u0026rsquo;s keys and values a one-time cost, separate from the per-step decoder cache — a clean computational separation the concatenation approach blurs.\n(c) Decoupled lengths and roles. The encoder\u0026rsquo;s depth, width, and even vocabulary can differ from the decoder\u0026rsquo;s; the two stacks specialize, one as a reader and one as a writer. The cost is the extra cross-attention sublayer and a more complex architecture — which is the entire subject of Lessons 4–5.\nInvariant (the whole course in one line). The encoder compresses context into $H_{\\text{enc}}$; the decoder generates $y$ conditioned on it; cross-attention is the only mechanism that did not already exist in a decoder-only model. If you internalize one sentence, make it this one.\nFigure 2. The two-stack layout, and the three things it buys.\n4. The function signature and the three number-categories Formally, an encoder–decoder model is a pair of maps sharing the parameter vector $\\theta\\in\\mathbb R^P$:\n$$ \\underbrace{\\mathrm{Enc}:\\ \\mathcal V_x^{\\le n}\\times\\Theta \\to \\mathbb R^{n\\times d}}_{x\\,\\mapsto\\,H_{\\text{enc}}}, \\qquad \\underbrace{\\mathrm{Dec}:\\ \\mathcal V_y^{\\lt t}\\times \\mathbb R^{n\\times d}\\times\\Theta \\to \\Delta^{V_y-1}}_{(y_{\\lt t},\\,H_{\\text{enc}})\\,\\mapsto\\,\\hat{\\mathbf p}_t}, $$where the decoder emits a categorical distribution $\\hat{\\mathbf p}_t$ over the output vocabulary on the probability simplex $\\Delta^{V_y-1}$. The encoder is evaluated once; the decoder is evaluated $m$ times, each time reading the same $H_{\\text{enc}}$ and one more output token. The parameter vector $\\theta$ now contains three groups instead of the decoder-only model\u0026rsquo;s one: encoder parameters, decoder parameters (which include the extra cross-attention projections), and the shared/​tied embeddings.\nAs in the decoder-only course, every number in this course belongs to exactly one category:\n[LEARNED] — $\\theta$: encoder and decoder weight matrices, the cross-attention projections $W_Q^{\\text{cross}},W_K^{\\text{cross}},W_V^{\\text{cross}}$, embeddings, the output head. [CONFIG] — architecture and process: encoder/decoder depths $N_{\\text{enc}},N_{\\text{dec}}$, width $d$, head count $h$, the masking patterns, the pretraining objective\u0026rsquo;s corruption rate, beam width. [STATE] — transient: $H_{\\text{enc}}$ itself (a function of the current input!), the decoder self-attention KV cache, the cross-attention KV cache, logits, attention matrices. Note one subtlety that recurs throughout: $H_{\\text{enc}}$ is [STATE], not [LEARNED]. It looks like a fixed thing during generation, but it is recomputed for every new input — it is the encoder\u0026rsquo;s activations, not its weights. The encoder\u0026rsquo;s weights are [LEARNED]; the memory it produces is [STATE].\nFigure 3. One encode, m decodes — and what each arrow is made of.\n5. What is genuinely new — and what is not It is worth drawing the line now, because the rest of the course lives on it. From the decoder-only model you already have, an encoder–decoder transformer reuses unchanged: token embeddings, positional information, scaled dot-product self-attention, multi-head structure, the FFN/MLP sublayer, residual connections and normalization, the softmax output head, and cross-entropy training with teacher forcing. It adds exactly one new operation — cross-attention (Lesson 4) — and rearranges masking (the encoder drops the causal mask; the decoder keeps it) and the training objective (denoising pretraining instead of plain next-token prediction; Lesson 7). That is the entire delta. The lessons ahead are mostly about getting those three changes exactly right.\nConclusion A sequence-to-sequence task asks for a conditional distribution $P_\\theta(y\\mid x)=\\prod_t P_\\theta(y_t\\mid y_{\\lt t},x)$ in which the input $x$ is fully available and the output $y$ is generated left to right. That asymmetry motivates two stacks: a bidirectional encoder that reads $x$ once into a memory $H_{\\text{enc}}\\in\\mathbb R^{n\\times d}$, and a causal decoder that generates $y$ while attending to that memory through cross-attention. Encoder-only models read but do not generate; decoder-only models generate but read causally and implicitly; the encoder–decoder keeps the two roles separate. Everything in the architecture is inherited from the decoder-only transformer except one new mechanism (cross-attention), one rearrangement of masks, and one change of objective. And $H_{\\text{enc}}$, the heart of the conditioning, is [STATE] — recomputed per input, never a parameter.\nWe open the box bottom-up, starting with the reader. Lesson 2 builds the encoder stack and shows what dropping the causal mask actually buys.\n← Previous: (none — this is the first lesson) · Course index · Next: Lesson 2 — The Encoder Stack → Appendices: A — Formulas · B — Architecture Comparison · C — Acronyms The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/tutorials/encoder-decoder-lessons/lesson-01-why-two-stacks/","summary":"\u003ch1 id=\"lesson-1--why-two-stacks-the-sequence-to-sequence-problem\"\u003eLesson 1 — Why Two Stacks? The Sequence-to-Sequence Problem\u003c/h1\u003e\n\u003cp\u003e\u003cem\u003eA 10-lesson technical course on encoder–decoder (seq2seq) transformers.\u003c/em\u003e\n\u003cem\u003eAudience: you already know the decoder-only transformer — self-attention, next-token prediction, softmax cross-entropy — and want the encoder–decoder nuances. Prerequisites: linear algebra, multivariate calculus, basic probability. We reuse standard notation and define what is new as we go.\u003c/em\u003e\u003c/p\u003e\n\u003chr\u003e\n\u003ch2 id=\"course-table-of-contents\"\u003eCourse table of contents\u003c/h2\u003e\n\u003ctable\u003e\n  \u003cthead\u003e\n      \u003ctr\u003e\n          \u003cth\u003e#\u003c/th\u003e\n          \u003cth\u003eLesson\u003c/th\u003e\n          \u003cth\u003eCore technical content\u003c/th\u003e\n      \u003c/tr\u003e\n  \u003c/thead\u003e\n  \u003ctbody\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e1\u003c/td\u003e\n          \u003ctd\u003e\u003cstrong\u003eWhy Two Stacks?\u003c/strong\u003e \u003cem\u003e(this file)\u003c/em\u003e\u003c/td\u003e\n          \u003ctd\u003eThe conditional map $P_\\theta(y\\mid x)$; encoder-only vs. decoder-only vs. encoder–decoder; why seq2seq wants two stacks\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e2\u003c/td\u003e\n          \u003ctd\u003e\u003ca href=\"/tutorials/encoder-decoder-lessons/lesson-02-the-encoder-stack/\"\u003eThe Encoder Stack\u003c/a\u003e\n\u003c/td\u003e\n          \u003ctd\u003eBidirectional self-attention (no mask); $H_{\\text{enc}}\\in\\mathbb R^{n\\times d}$ as fixed memory; padding masks; vs. BERT\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e3\u003c/td\u003e\n          \u003ctd\u003e\u003ca href=\"/tutorials/encoder-decoder-lessons/lesson-03-the-decoder-stack/\"\u003eThe Decoder Stack\u003c/a\u003e\n\u003c/td\u003e\n          \u003ctd\u003eCausal self-attention revisited; the three-sublayer block; what is added vs. a decoder-only model\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e4\u003c/td\u003e\n          \u003ctd\u003e\u003ca href=\"/tutorials/encoder-decoder-lessons/lesson-04-cross-attention/\"\u003eCross-Attention\u003c/a\u003e\n\u003c/td\u003e\n          \u003ctd\u003e$Q=H_{\\text{dec}}W_Q$, $K,V=H_{\\text{enc}}W_{K},H_{\\text{enc}}W_V$; the non-square $m\\times n$ map; gradient into the encoder\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e5\u003c/td\u003e\n          \u003ctd\u003e\u003ca href=\"/tutorials/encoder-decoder-lessons/lesson-05-the-encoder-decoder-block/\"\u003eThe Encoder–Decoder Block\u003c/a\u003e\n\u003c/td\u003e\n          \u003ctd\u003eSelf-attn + cross-attn + FFN; full architecture; parameter budget; T5 vs. BART\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e6\u003c/td\u003e\n          \u003ctd\u003e\u003ca href=\"/tutorials/encoder-decoder-lessons/lesson-06-prefix-lm/\"\u003ePrefix-LM\u003c/a\u003e\n\u003c/td\u003e\n          \u003ctd\u003eThe prefix mask; folding two stacks into one; the encoder-only ↔ decoder-only spectrum\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e7\u003c/td\u003e\n          \u003ctd\u003e\u003ca href=\"/tutorials/encoder-decoder-lessons/lesson-07-pretraining-objectives/\"\u003ePretraining Objectives\u003c/a\u003e\n\u003c/td\u003e\n          \u003ctd\u003eT5 span corruption with sentinels; BART\u0026rsquo;s five noises; vs. MLM and next-token CE\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e8\u003c/td\u003e\n          \u003ctd\u003e\u003ca href=\"/tutorials/encoder-decoder-lessons/lesson-08-training-teacher-forcing/\"\u003eTraining: Teacher Forcing\u003c/a\u003e\n\u003c/td\u003e\n          \u003ctd\u003eShifted decoder inputs; CE over target positions; parallel training; gradient through cross-attention; exposure bias\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e9\u003c/td\u003e\n          \u003ctd\u003e\u003ca href=\"/tutorials/encoder-decoder-lessons/lesson-09-inference/\"\u003eInference\u003c/a\u003e\n\u003c/td\u003e\n          \u003ctd\u003eEncode-once/decode-many; the two KV caches; beam search; coverage and copy mechanisms\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e10\u003c/td\u003e\n          \u003ctd\u003e\u003ca href=\"/tutorials/encoder-decoder-lessons/lesson-10-finetuning-and-distillation/\"\u003eFine-Tuning \u0026amp; Distillation\u003c/a\u003e\n\u003c/td\u003e\n          \u003ctd\u003eText-to-text framing; translation/summarization; BLEU/ROUGE; distilling T5-small\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eA\u003c/td\u003e\n          \u003ctd\u003e\u003ca href=\"/tutorials/encoder-decoder-lessons/appendix-a-formulas/\"\u003eAppendix A — Formulas\u003c/a\u003e\n\u003c/td\u003e\n          \u003ctd\u003eAll key equations, collected by lesson\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eB\u003c/td\u003e\n          \u003ctd\u003e\u003ca href=\"/tutorials/encoder-decoder-lessons/appendix-b-architecture-comparison/\"\u003eAppendix B — Architecture Comparison\u003c/a\u003e\n\u003c/td\u003e\n          \u003ctd\u003eThe three families along masking, conditioning, objective, use-case axes\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eC\u003c/td\u003e\n          \u003ctd\u003e\u003ca href=\"/tutorials/encoder-decoder-lessons/appendix-c-acronyms/\"\u003eAppendix C — Acronyms\u003c/a\u003e\n\u003c/td\u003e\n          \u003ctd\u003eEvery acronym and symbol, expanded\u003c/td\u003e\n      \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp\u003eOne invariant runs through every lesson and is worth stating before anything else:\u003c/p\u003e","title":"Why Two Stacks? The Sequence-to-Sequence Problem"},{"content":"Lesson 2 — From the Dense FFN to a Layer of Experts Technical course on MoE LLM mechanics — Lesson 2 of 10.\nRecap of Lesson 1 A Mixture-of-Experts model is a dense LLM with one component changed: the per-block FFN becomes $E$ expert FFNs plus a router that selects $k\\ll E$ of them per token, combining their outputs as $\\mathrm{MoE}(\\mathbf x)=\\sum_{i\\in\\mathcal T_k(\\mathbf x)}g_i(\\mathbf x)E_i(\\mathbf x)$. This splits the single parameter count into total $P_{\\text{tot}}$ (capacity, memory) and activated $P_{\\text{act}}$ (per-token compute), with compute $\\approx 6P_{\\text{act}}D$ but memory $\\propto P_{\\text{tot}}$. We now build the layer itself, starting from the exact component it replaces and counting precisely how the parameter budget changes.\nLesson 1 said what an MoE layer computes. This lesson says what it is made of: we take the dense FFN the companion course already dissected, stamp out $E$ copies of it, and ask where this new block sits in the stack and what it does to the parameter count. The router that chooses among the copies is deferred to Lesson 3; here the experts are our subject.\n1. The component being replaced Recall the dense position-wise FFN, applied identically and independently to each token vector $\\mathbf x\\in\\mathbb R^d$:\n$$ \\mathrm{FFN}(\\mathbf x)=W_2\\,\\phi\\!\\big(W_1\\mathbf x+\\mathbf b_1\\big)+\\mathbf b_2,\\qquad W_1\\in\\mathbb R^{d_{ff}\\times d},\\ W_2\\in\\mathbb R^{d\\times d_{ff}}, $$with hidden width $d_{ff}=4d$ and a nonlinearity $\\phi$ (GELU, or a SwiGLU gated unit). Its parameter count is $\\approx 2\\,d\\,d_{ff}=8d^2$, versus attention\u0026rsquo;s $\\approx 4d^2$ — so the FFN is about two-thirds of each block\u0026rsquo;s parameters, and across a deep model the FFNs hold the majority of the weights and thus most of the model\u0026rsquo;s stored knowledge.\nThis last fact is why MoE targets the FFN and not attention. If you want to grow capacity cheaply, you grow the part that holds the capacity — and you grow it in a way that does not grow per-token compute. Replicating the FFN into experts does exactly that.\n2. The expert layer, defined An expert $E_i$ is just an ordinary FFN with its own independent weights:\n$$ E_i(\\mathbf x)=W_2^{(i)}\\,\\phi\\!\\big(W_1^{(i)}\\mathbf x+\\mathbf b_1^{(i)}\\big)+\\mathbf b_2^{(i)},\\qquad i=1,\\dots,E. $$All $E$ experts share the same shape (width $d$, hidden $d_{ff}$, nonlinearity $\\phi$) but have different learned weights — exactly the relationship the $N$ transformer blocks have to one another. There is nothing special about an expert\u0026rsquo;s internals; an expert is a normal FFN. What is special is that the model holds $E$ of them per MoE layer and runs only $k$.\nGiven a router that outputs a sparse gate vector $\\mathbf g(\\mathbf x)\\in\\mathbb R^E$ (Lesson 3) with support on the top-$k$ indices $\\mathcal T_k(\\mathbf x)$, the MoE layer is\n$$ \\boxed{\\ \\mathrm{MoE}(\\mathbf x)=\\sum_{i\\in\\mathcal T_k(\\mathbf x)} g_i(\\mathbf x)\\,E_i(\\mathbf x)\\ } $$It drops straight into the transformer block in place of the FFN, keeping the same pre-norm residual wrapper from the dense course:\n$$ \\begin{aligned} \\tilde H \u0026= H + \\mathrm{MHA}\\big(\\mathrm{Norm}(H)\\big),\\\\ H' \u0026= \\tilde H + \\mathrm{MoE}\\big(\\mathrm{Norm}(\\tilde H)\\big). \\end{aligned} $$Attention, residuals, and normalization are untouched — this is the entire point of Lesson 1\u0026rsquo;s \u0026ldquo;one component changed.\u0026rdquo; Every difference in the rest of the course flows from this single substitution in the second sub-layer.\nFigure 1. The MoE layer drops into the pre-norm block, in place of the FFN.\n3. The parameter explosion (and why it is the goal) Replacing one FFN with $E$ of them multiplies the FFN parameter budget by $E$. Per MoE layer, the experts hold\n$$ P_{\\text{experts/layer}}\\approx E\\times 2dd_{ff}=8Ed^2, $$against attention\u0026rsquo;s $\\approx 4d^2$ and the router\u0026rsquo;s tiny $Ed$ (Lesson 3). For $E=8$ this makes the layer\u0026rsquo;s parameters roughly $\\tfrac{8\\cdot 8d^2}{8\\cdot 8d^2+4d^2}\\approx 94\\%$ experts; for $E=64$ it is over $99\\%$. In an MoE model, essentially all the parameters live in the experts. That is the parameter explosion — and it is deliberate. It is how $P_{\\text{tot}}$ grows.\nBut per-token compute does not explode, because only $k$ experts run. The activated expert parameters per token are\n$$ P_{\\text{act,experts/layer}}\\approx k\\times 2dd_{ff}=8kd^2, $$independent of $E$. So increasing $E$ at fixed $k$ raises $P_{\\text{tot}}$ (and memory) while leaving $P_{\\text{act}}$ (and FLOPs) flat. Concretely: Mixtral-8×7B has $E=8,\\ k=2$. Naively \u0026ldquo;$8\\times7\\mathrm B = 56\\mathrm B$,\u0026rdquo; but the experts overlap with shared attention/embedding weights, so $P_{\\text{tot}}\\approx 47\\mathrm B$; with $k=2$ of $8$ experts active, $P_{\\text{act}}\\approx 13\\mathrm B$. The model has the memory of a 47B model and the FLOPs of a 13B one. (The name \u0026ldquo;8×7B\u0026rdquo; is marketing arithmetic, not the real count — Lesson 1\u0026rsquo;s two-number lesson in one example.)\nContrast with the dense model. Growing a dense model means widening $d$ or deepening $N$ — every new parameter is paid for on every token, so capacity and cost rise together. Growing an MoE means raising $E$ — new parameters are paid for only on the tokens routed to them, so capacity rises and per-token cost stays flat. Dense scaling moves both columns of the Lesson 1 table; MoE scaling moves only the memory column.\nFigure 2. Where the parameters go, and how few of them run.\n4. Where the experts go: placement and granularity Two design choices shape how the expert idea is deployed across the model.\nWhich layers are MoE? You need not make every block an MoE block. Common choices:\nEvery block an MoE layer (Switch Transformer style), or Every other block an MoE layer, with the rest left as ordinary dense FFNs (GShard, many GLaM-style models), which halves the expert-memory burden and the routing/communication overhead while keeping most of the capacity benefit. Whether a given block is MoE or dense is a [CONFIG] choice. Dense blocks behave exactly as in the companion course; only the MoE blocks route.\nGranularity: how big is an expert? The original recipe gives each expert the full FFN width $d_{ff}=4d$. Fine-grained experts (DeepSeek-MoE) instead make each expert narrower — say $d_{ff}/m$ — and use $m$ times as many of them, selecting proportionally more. Splitting $E$ standard experts into $mE$ slim ones and raising $k$ to $mk$ keeps $P_{\\text{tot}}$ and $P_{\\text{act}}$ unchanged but vastly increases the number of combinations $\\binom{mE}{mk}$ the router can express, giving finer specialization. Granularity ($m$) is [CONFIG].\nShared (always-on) experts A complementary idea (also DeepSeek-MoE) keeps a small number of shared experts that every token passes through, alongside the routed ones:\n$$ \\mathrm{MoE}(\\mathbf x)=\\underbrace{\\sum_{j\\in\\mathcal S} E_j(\\mathbf x)}_{\\text{shared, always on}}+\\underbrace{\\sum_{i\\in\\mathcal T_k(\\mathbf x)} g_i(\\mathbf x)\\,E_i(\\mathbf x)}_{\\text{routed}}. $$The shared experts absorb the common computation every token needs (so the routed experts need not each re-learn it), which improves parameter efficiency and stability. Shared experts add to $P_{\\text{act}}$ for every token — they are activated unconditionally — so they are a deliberate, accounted-for cost. The count $|\\mathcal S|$ is [CONFIG].\nFigure 3. Three CONFIG knobs on top of E and k.\n5. Learned / config / state Quantity Category Expert weights $W_1^{(i)},W_2^{(i)},\\mathbf b_1^{(i)},\\mathbf b_2^{(i)}$, all $i$ [LEARNED] Router matrix $W_g$ (built in Lesson 3) [LEARNED] $E$, $k$, granularity $m$, shared-expert count $ \\mathcal S Gate vector $\\mathbf g(\\mathbf x)$, selected set $\\mathcal T_k(\\mathbf x)$, expert hidden activations [STATE] Conclusion An MoE layer is built by replicating the dense FFN $E$ times into experts $E_i(\\mathbf x)=W_2^{(i)}\\phi(W_1^{(i)}\\mathbf x+\\mathbf b_1^{(i)})+\\mathbf b_2^{(i)}$ and combining a router-selected $k$ of them as $\\mathrm{MoE}(\\mathbf x)=\\sum_{i\\in\\mathcal T_k(\\mathbf x)}g_i(\\mathbf x)E_i(\\mathbf x)$, dropped into the same pre-norm residual block in place of the FFN. Because the FFN already held $\\approx\\tfrac23$ of the dense parameters, replicating it makes the experts hold essentially all of an MoE\u0026rsquo;s parameters: $P_{\\text{tot}}$ grows with $E$ while $P_{\\text{act}}$ stays pinned to $k$ — capacity up, per-token cost flat. Design knobs are which layers are MoE, expert granularity (slim experts, more of them, finer specialization), and shared experts (always-on, absorbing common computation). Attention, residuals, and norm are untouched; the whole MoE story lives in this one sub-layer.\nWe have $E$ experts but no way yet to choose among them. The next lesson builds the router — the small learned network that, per token, decides who works.\n← Lesson 1 — The Big Picture · Course index · Next: Lesson 3 — The Router / Gating Network → Appendices: A — MoE Family · B — Dense vs. MoE · C — Acronyms The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/tutorials/moe-lessons/lesson-02-from-ffn-to-experts/","summary":"\u003ch1 id=\"lesson-2--from-the-dense-ffn-to-a-layer-of-experts\"\u003eLesson 2 — From the Dense FFN to a Layer of Experts\u003c/h1\u003e\n\u003cp\u003e\u003cem\u003eTechnical course on MoE LLM mechanics — Lesson 2 of 10.\u003c/em\u003e\u003c/p\u003e\n\u003chr\u003e\n\u003ch2 id=\"recap-of-lesson-1\"\u003eRecap of Lesson 1\u003c/h2\u003e\n\u003cp\u003eA Mixture-of-Experts model is a dense LLM with one component changed: the per-block FFN becomes $E$ expert FFNs plus a router that selects $k\\ll E$ of them per token, combining their outputs as $\\mathrm{MoE}(\\mathbf x)=\\sum_{i\\in\\mathcal T_k(\\mathbf x)}g_i(\\mathbf x)E_i(\\mathbf x)$. This splits the single parameter count into \u003cstrong\u003etotal\u003c/strong\u003e $P_{\\text{tot}}$ (capacity, memory) and \u003cstrong\u003eactivated\u003c/strong\u003e $P_{\\text{act}}$ (per-token compute), with compute $\\approx 6P_{\\text{act}}D$ but memory $\\propto P_{\\text{tot}}$. We now build the layer itself, starting from the exact component it replaces and counting precisely how the parameter budget changes.\u003c/p\u003e","title":"From the Dense FFN to a Layer of Experts"},{"content":"Chapter 2 — From Words to Numbers In the last chapter we settled what a model is: a function, set by billions of learned knobs, that turns an input into an output. But there is a gap in that picture we glossed over. A computer, at bottom, only ever does arithmetic. It manipulates numbers. And yet we hand it words — fuzzy, ambiguous, deeply human things. Something has to bridge that gap before any of the clever machinery can begin. This chapter is about that bridge. It comes in two steps: first chopping text into pieces, and then turning each piece into a point on a vast \u0026ldquo;map of meaning.\u0026rdquo;\nStep one: chopping text into tokens The first thing that happens to your text, before the model does anything intelligent with it, is that it gets cut into chunks. These chunks are called tokens. You might expect a token to be a word, but it usually isn\u0026rsquo;t quite. Tokens tend to be common word-fragments. The word \u0026ldquo;tokenization\u0026rdquo; might be split into \u0026ldquo;token\u0026rdquo; and \u0026ldquo;ization.\u0026rdquo; Frequent short words like \u0026ldquo;the\u0026rdquo; stay whole; rarer or longer words get broken into familiar pieces.\nWhy pieces rather than whole words? It is a practical compromise between two bad extremes. If you insisted on whole words only, the model would be helpless the first time it met a word that wasn\u0026rsquo;t on its list — a new name, a typo, a piece of slang, a technical term. There are endlessly many possible words, and no fixed list can hold them all. But if you went to the opposite extreme and fed the model individual letters, every sentence would become an enormously long string of tiny pieces, and the model would drown in length. Word-fragments thread the needle: common words stay efficient and whole, anything unusual breaks down into known fragments, and as a last resort literally anything can be spelled out letter by letter. The model is never stumped by an unfamiliar word; it just assembles it from smaller parts.\nThis chopping is done by a fixed recipe built once, in advance. It never learns and never changes. It is a dumb, mechanical lookup — and at this stage each token is nothing but an identification number, like a coat-check ticket. The number means nothing in itself. \u0026ldquo;Token number 4,219\u0026rdquo; carries no meaning yet; it is just a stub you can redeem for the real thing in the next step.\nIt is worth knowing that this innocent chopping step is the source of some of the famous quirks of language models. When a model fumbles simple arithmetic, or miscounts the letters in a word, it is often because the chopping did not line up with the digits or letters in the way a human would. The model never saw the word the way you see it; it saw a handful of fragments.\nFigure 1. Three ways to chop up text, and the one that shipped.\nStep two: turning tokens into points on a map of meaning Here is where it gets beautiful. Each token\u0026rsquo;s ID number is looked up in a giant table — itself one of the learned things, written during training — that converts the number into a long list of numbers. That list is the token\u0026rsquo;s embedding. The intuition you want is this: each token is being placed at a specific location in an enormous, many-dimensioned \u0026ldquo;map of meaning.\u0026rdquo;\nOn this map, location encodes meaning. Words that mean similar things end up near one another; words used in very different ways end up far apart. \u0026ldquo;King,\u0026rdquo; \u0026ldquo;queen,\u0026rdquo; and \u0026ldquo;prince\u0026rdquo; settle into one neighborhood. \u0026ldquo;Tuesday\u0026rdquo; and \u0026ldquo;Wednesday\u0026rdquo; cluster in another, across the map. Nobody assigns these locations by hand. The geometry emerges during training, for the simple reason that placing similar words near each other helps the model predict well — and predicting well is the only thing training rewards.\nThe genuinely surprising part is that directions on this map turn out to carry meaning too, not just neighborhoods. The step that takes you from \u0026ldquo;man\u0026rdquo; to \u0026ldquo;woman\u0026rdquo; is roughly the same step, in the same direction, that takes you from \u0026ldquo;king\u0026rdquo; to \u0026ldquo;queen.\u0026rdquo; Move from a country to its capital and the displacement is consistent. This is the celebrated observation that, on the map of meaning, \u0026ldquo;king minus man plus woman lands you near queen.\u0026rdquo; It is not programmed. It falls out, unbidden, from the training process — meaning quietly reorganizing itself into geometry.\nThis is the deeper reason embeddings matter so much. They are how raw symbols become semantic. Once a word is a location on a map where nearness means similarity and direction means relationship, all the later machinery has something rich to work with. The arithmetic the computer is so good at can now operate on meaning, because meaning has been turned into coordinates.\nFigure 2. From a coat-check ticket to a location on the map of meaning.\nThe problem of order There is one thing a location on the map of meaning does not capture: where the word sat in the sentence. And order plainly matters. \u0026ldquo;Dog bites man\u0026rdquo; and \u0026ldquo;man bites dog\u0026rdquo; are built from the very same words, yet they describe opposite events. If all the model had were the meaning-coordinates of the words, with no sense of their arrangement, it could not tell these apart.\nSo alongside each word\u0026rsquo;s meaning, the model is also given information about its position. The clever modern approach does not just stamp each word with an absolute seat number (\u0026ldquo;you are word number seven\u0026rdquo;). Instead it encodes relative distance — how far apart any two words are from each other. This turns out to be what actually matters for understanding language, and it has a happy side effect: a model built around relative distances handles longer texts more gracefully than one trained to recognize fixed seat numbers it may never have seen during training. The names attached to these schemes can get baroque, but the idea is plain: the model needs to know not only what each word means but where it stands in relation to the others.\nPutting the two steps together Let us trace a sentence through the bridge. You type a sentence. First it is chopped into tokens — mostly familiar word-fragments, each now just an ID number. Then each ID is looked up and placed at its location on the map of meaning, and stamped with information about its position in the sentence. What was a string of human characters is now a sequence of rich numerical points, each one carrying both what the word means and where it sits. The fuzzy human thing has become something the model\u0026rsquo;s arithmetic can grip.\nAnd yet something is still missing, and it is the most important thing of all. Right now, each word\u0026rsquo;s meaning-coordinate was computed in isolation, by simple lookup. The word \u0026ldquo;bank\u0026rdquo; gets the same starting location whether the sentence is about a river or about money. The word \u0026ldquo;it\u0026rdquo; arrives with no idea what it refers to. Each word, so far, is an island. Before the model can understand anything, the words have to talk to each other — to look around, figure out which of their neighbors matter, and revise their own meaning in light of the company they keep.\nThat conversation among words is the mechanism at the very heart of every model in this book. It is called attention, and it is the subject of Chapter 3 .\nFigure 3. The whole bridge, and the thing still missing at the end of it.\n← Chapter 1 · What These Machines Are · Contents · Chapter 3 · Attention → The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/tutorials/inside-the-machines/chapter-02-from-words-to-numbers/","summary":"\u003ch1 id=\"chapter-2--from-words-to-numbers\"\u003eChapter 2 — From Words to Numbers\u003c/h1\u003e\n\u003cp\u003eIn the last chapter we settled what a model is: a function, set by billions of learned knobs, that turns an input into an output. But there is a gap in that picture we glossed over. A computer, at bottom, only ever does arithmetic. It manipulates numbers. And yet we hand it \u003cem\u003ewords\u003c/em\u003e — fuzzy, ambiguous, deeply human things. Something has to bridge that gap before any of the clever machinery can begin. This chapter is about that bridge. It comes in two steps: first chopping text into pieces, and then turning each piece into a point on a vast \u0026ldquo;map of meaning.\u0026rdquo;\u003c/p\u003e","title":"From Words to Numbers"},{"content":"Lesson 2 — The Encoder Stack: Bidirectional Context Technical course on encoder–decoder transformers — Lesson 2 of 10.\nRecap of Lesson 1 A seq2seq model is the conditional map $P_\\theta(y\\mid x)=\\prod_t P_\\theta(y_t\\mid y_{\\lt t},x)$: the input $x$ is fully available, the output $y$ is generated left to right. That asymmetry motivates two stacks — a bidirectional encoder that reads $x$ once into a memory $H_{\\text{enc}}\\in\\mathbb R^{n\\times d}$ (which is [STATE], recomputed per input), and a causal decoder that generates conditioned on it via cross-attention, the one genuinely new mechanism. We now build the first stack and make precise what dropping the causal mask buys.\n1. The encoder\u0026rsquo;s one job: compress $x$ into a memory The encoder is a reader. It takes the input token sequence $x=(x_1,\\dots,x_n)$, embeds it into $H^{(0)}\\in\\mathbb R^{n\\times d}$ exactly as a decoder-only model would (token embedding $E_x\\in\\mathbb R^{V_x\\times d}$ plus positional information), and runs it through $N_{\\text{enc}}$ transformer blocks to produce\n$$ H_{\\text{enc}}:=H^{(N_{\\text{enc}})}\\in\\mathbb R^{n\\times d}. $$That is the entire output of the encoder — one contextual vector per input token, not a single pooled vector. This matters: the decoder will attend over all $n$ of these rows individually (Lesson 4), so the \u0026ldquo;compression\u0026rdquo; is not into a fixed-size bottleneck but into a length-$n$ set of context-rich vectors. (The single-vector bottleneck was exactly the limitation of pre-attention RNN seq2seq models that attention was invented to remove.) Once produced, $H_{\\text{enc}}$ is frozen for the whole of generation — it is computed in a single forward pass and then only read.\n2. Bidirectional self-attention: the missing mask Here is the one structural difference between an encoder block and the decoder-only block you know. Recall scaled dot-product attention:\n$$ \\mathrm{Attn}(Q,K,V)=\\mathrm{softmax}\\!\\Big(\\tfrac{QK^\\top}{\\sqrt{d_k}}+M\\Big)V, \\qquad Q=HW_Q,\\ K=HW_K,\\ V=HW_V . $$In a decoder-only model the mask $M$ is the $-\\infty$ upper-triangular causal mask: position $i$ may not attend to $j\\gt i$. The encoder simply sets $M=0$ (apart from padding, §4). Every input token attends to every other — past and future:\n$$ M^{\\text{enc}}_{ij}=0\\ \\ \\forall i,j \\quad\\Longrightarrow\\quad A_{ij}\\gt 0\\ \\text{for all } i,j,\\qquad A\\mathbf 1=\\mathbf 1. $$The attention matrix $A\\in\\mathbb R^{n\\times n}$ is now full (dense, row-stochastic) rather than lower-triangular. Mechanically the change is trivial — delete the mask. Conceptually it is the reason the encoder exists. Because the input is known in full before any output token is produced, there is no notion of \u0026ldquo;future leakage\u0026rdquo; to prevent: letting token $i$ see token $j\\gt i$ does not cheat any prediction, because the encoder predicts nothing. It only represents.\nWhy this helps is concrete. Consider encoding the English source \u0026ldquo;the bank of the river\u0026rdquo; for translation. The correct sense of \u0026ldquo;bank\u0026rdquo; is fixed by \u0026ldquo;river,\u0026rdquo; which appears after it. A causal reader processing \u0026ldquo;the bank\u0026rdquo; has not yet seen \u0026ldquo;river\u0026rdquo; and must commit to a representation prematurely; the bidirectional encoder lets \u0026ldquo;bank\u0026rdquo; attend forward to \u0026ldquo;river\u0026rdquo; and resolve the ambiguity in the same layer. Every downstream cross-attention then reads a \u0026ldquo;bank\u0026rdquo; vector that already knows it is a riverbank.\nflowchart TB subgraph causal[\"Decoder-only self-attn (causal)\"] direction LR d1[\"t1\"] --\u003e d2[\"t2\"] --\u003e d3[\"t3\"] end subgraph bidir[\"Encoder self-attn (bidirectional)\"] direction LR e1[\"t1\"] \u003c--\u003e e2[\"t2\"] \u003c--\u003e e3[\"t3\"] e1 \u003c--\u003e e3 end Everything else inside the block — the $\\sqrt{d_k}$ scaling (the same variance argument: raw logits have standard deviation $\\sqrt{d_k}$, so we rescale to keep softmax out of its zero-gradient saturation regime), the softmax Jacobian $\\mathrm{diag}(\\mathbf a)-\\mathbf a\\mathbf a^\\top$, multi-head structure with $d_k=d/h$ — is identical to what you already know. Only $M$ changed.\nFigure 1. The missing mask, drawn.\n3. The encoder block One encoder block is two pre-norm residual sublayers — multi-head bidirectional self-attention and a position-wise FFN — exactly the standard transformer block minus the causal mask:\n$$ \\begin{aligned} H' \u0026= H + \\mathrm{MHA}_{\\text{bidir}}\\big(\\mathrm{Norm}(H)\\big),\\\\ H^{\\text{out}} \u0026= H' + \\mathrm{FFN}\\big(\\mathrm{Norm}(H')\\big), \\end{aligned} \\qquad \\mathrm{FFN}(\\mathbf x)=W_2\\,\\phi(W_1\\mathbf x+\\mathbf b_1)+\\mathbf b_2,\\ \\ d_{ff}\\approx 4d . $$The residual gives the by-now-familiar layer Jacobian $I+\\partial\\mathcal F$ — an identity gradient path that makes deep stacks trainable. Stacking $N_{\\text{enc}}$ such blocks yields $H_{\\text{enc}}$. The parameter budget per encoder block is the usual $\\approx 4d^2$ (attention: $W_Q,W_K,W_V,W_O$) $+\\ \\approx 8d^2$ (FFN) $=12d^2$; there is no cross-attention here — cross-attention lives only in the decoder.\nNote (what an encoder block is not). It is not a \u0026ldquo;smaller decoder.\u0026rdquo; It has two sublayers, not three; it never sees the output sequence; and it produces representations, not a probability distribution. There is no output head on the encoder of a standard T5/BART model.\nFigure 2. The encoder block: two sublayers, ≈ 12d² parameters.\n4. Padding masks: the mask that does survive We said the encoder drops the causal mask. It keeps a different one. Real batches contain sequences of different lengths $n$, padded to a common length with a special \u0026lt;pad\u0026gt; token. Those pad positions carry no information and must not be attended to, or they would leak meaningless vectors into every real token\u0026rsquo;s representation. So the encoder uses a padding mask: for each query $i$, any key position $j$ that is padding gets $M_{ij}=-\\infty$.\n$$ M^{\\text{pad}}_{ij}=\\begin{cases}0,\u0026 x_j\\neq\\texttt{\\textless{}pad\\textgreater{}}\\\\ -\\infty,\u0026 x_j=\\texttt{\\textless{}pad\\textgreater{}}\\end{cases} $$This is [CONFIG] in the sense of being a deterministic function of the input lengths, with zero parameters — but unlike the causal mask it depends on the data (which positions are padding), so in practice it travels with the batch as [STATE]. The same encoder padding mask is reused later by the decoder\u0026rsquo;s cross-attention (Lesson 4), so that generated tokens never attend to padded input positions. Keep it in mind: it is the one mask the encoder carries forward into the rest of the model.\n5. Is the encoder just BERT? Architecturally, an encoder stack with bidirectional self-attention is essentially BERT\u0026rsquo;s stack — and that is a useful anchor. But two differences matter for our story:\nNo MLM head, no [CLS] objective. BERT is trained with a masked-language-model head and used standalone for classification via a pooled [CLS] vector. The T5/BART encoder has no such head; it is trained jointly with the decoder on a seq2seq reconstruction objective (Lesson 7), and its output $H_{\\text{enc}}$ exists only to be consumed by cross-attention. The encoder is a component, not a finished model. It produces all $n$ vectors for attention, not a pooled summary. Sentence-embedding uses of BERT often pool $H$ to one vector; the encoder–decoder keeps all $n$ rows precisely so the decoder can attend to different source positions at different output steps (the alignment behavior of Lesson 9). So: same self-attention machinery as BERT, different role. The encoder is the half of a seq2seq model that does the \u0026ldquo;reading,\u0026rdquo; and it reads bidirectionally because it can.\nLearned / config / state Quantity Category Embedding $E_x$; per-block $W_Q,W_K,W_V,W_O,W_1,W_2,\\gamma,\\beta$ [LEARNED] Depth $N_{\\text{enc}}$, width $d$, heads $h$, $d_{ff}$, bidirectional ($M=0$) policy [CONFIG] Padding mask pattern (data-dependent) [CONFIG]/[STATE] Hidden states $H^{(\\ell)}$, attention matrices $A$, the memory $H_{\\text{enc}}$ [STATE] Figure 3. Same self-attention machinery, different role.\nConclusion The encoder is the reader. It embeds $x$ and runs $N_{\\text{enc}}$ standard transformer blocks whose only deviation from the decoder-only block is a deleted causal mask: self-attention is bidirectional ($M=0$), so every input token is contextualized by the whole input, past and future — the right thing to do when the input is fully known. It keeps a padding mask so pad tokens are never attended to, and it passes that mask downstream. Its output is the memory $H_{\\text{enc}}\\in\\mathbb R^{n\\times d}$: one contextual vector per input token, computed once and then frozen — and it is [STATE], the encoder\u0026rsquo;s activations, not its weights. Architecturally it is BERT\u0026rsquo;s stack, but it carries no output head and exists only to feed cross-attention.\nWe now have a reader. Next we build the writer — the decoder stack — and see what its causal self-attention shares with a decoder-only model, and what extra sublayer it grows to consume the memory we just built.\n← Lesson 1 — Why Two Stacks? · Course index · Next: Lesson 3 — The Decoder Stack → Appendices: A — Formulas · B — Architecture Comparison · C — Acronyms The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/tutorials/encoder-decoder-lessons/lesson-02-the-encoder-stack/","summary":"\u003ch1 id=\"lesson-2--the-encoder-stack-bidirectional-context\"\u003eLesson 2 — The Encoder Stack: Bidirectional Context\u003c/h1\u003e\n\u003cp\u003e\u003cem\u003eTechnical course on encoder–decoder transformers — Lesson 2 of 10.\u003c/em\u003e\u003c/p\u003e\n\u003chr\u003e\n\u003ch2 id=\"recap-of-lesson-1\"\u003eRecap of Lesson 1\u003c/h2\u003e\n\u003cp\u003eA seq2seq model is the conditional map $P_\\theta(y\\mid x)=\\prod_t P_\\theta(y_t\\mid y_{\\lt t},x)$: the input $x$ is fully available, the output $y$ is generated left to right. That asymmetry motivates \u003cstrong\u003etwo stacks\u003c/strong\u003e — a bidirectional \u003cstrong\u003eencoder\u003c/strong\u003e that reads $x$ once into a memory $H_{\\text{enc}}\\in\\mathbb R^{n\\times d}$ (which is \u003cstrong\u003e[STATE]\u003c/strong\u003e, recomputed per input), and a causal \u003cstrong\u003edecoder\u003c/strong\u003e that generates conditioned on it via \u003cstrong\u003ecross-attention\u003c/strong\u003e, the one genuinely new mechanism. We now build the first stack and make precise what \u003cem\u003edropping the causal mask\u003c/em\u003e buys.\u003c/p\u003e","title":"The Encoder Stack: Bidirectional Context"},{"content":"Lesson 2 — The Forward Process: Destroying Structure with Noise Recap of Lesson 1. A diffusion model is built on an asymmetry: a fixed forward process $q$ that gradually turns data $x_0$ into pure noise $x_T$, and a learned reverse process $p_\\theta$ that denoises step by step. Generation = running the learned reverse from noise. We met the single forward step $q(x_t\\mid x_{t-1}) = \\mathcal{N}(x_t; \\sqrt{1-\\beta_t}\\,x_{t-1}, \\beta_t I)$ and the assumed-Gaussian reverse step. Now we make the forward side completely rigorous.\nSuppose you needed to know what a photograph looks like after seven hundred specks of dust have settled on it. The honest method is to drop them one at a time and watch — seven hundred patient steps before you can even begin to work. It is precisely the kind of bookkeeping that makes a method too expensive to love. This lesson finds the shortcut: a single formula that hands you the seven-hundredth state directly, with no waiting in between. Almost everything cheap and practical about training a diffusion model traces back to this one act of teleportation.\nOverview Pouring sand one handful at a time has a cost: to know the photo after $700$ handfuls you would seemingly simulate all $700$ pourings in order, making every gradient step a long sequential simulation. The gift of the Gaussian forward process is that you can teleport: a closed-form expression gives the distribution of $x_t$ directly from $x_0$, for any $t$, with no intermediate simulation. Pick a random noise level, jump there, train. This single fact is what makes diffusion training cheap.\nThe forward process, formally The forward (diffusion) process is a Markov chain that injects Gaussian noise according to a variance schedule $\\beta_1,\\dots,\\beta_T$:\n$$ q(x_{1:T}\\mid x_0) = \\prod_{t=1}^{T} q(x_t \\mid x_{t-1}), \\qquad q(x_t \\mid x_{t-1}) = \\mathcal{N}\\!\\left(x_t;\\ \\sqrt{1-\\beta_t}\\,x_{t-1},\\ \\beta_t I\\right). $$Markov means each step depends only on the immediately previous state. There are no learnable parameters; $\\beta_t$ is fixed in advance. By the Gaussian reparameterization, drawing $x_t$ is the deterministic update\n$$ x_t = \\sqrt{1-\\beta_t}\\, x_{t-1} + \\sqrt{\\beta_t}\\,\\epsilon_{t-1}, \\qquad \\epsilon_{t-1}\\sim\\mathcal{N}(0,I). $$The factor $\\sqrt{1-\\beta_t}\\lt 1$ contracts the signal; the additive $\\sqrt{\\beta_t}\\,\\epsilon$ injects fresh noise. This balance keeps the total variance stable — formally, this is a variance-preserving (VP) process: if $\\mathrm{Var}(x_{t-1})=I$ then $\\mathrm{Var}(x_t)=(1-\\beta_t)I+\\beta_t I=I$.\nFigure 1. One forward step, and why the variance never moves.\nThe key trick: jumping to any $t$ in closed form Here is where the forward process stops being a chore and becomes a gift.\nIntroduce two shorthand quantities:\n$$ \\alpha_t := 1-\\beta_t, \\qquad \\bar{\\alpha}_t := \\prod_{s=1}^{t}\\alpha_s. $$Claim. $$ \\boxed{\\,q(x_t\\mid x_0) = \\mathcal{N}\\!\\left(x_t;\\ \\sqrt{\\bar\\alpha_t}\\,x_0,\\ (1-\\bar\\alpha_t) I\\right)\\,} \\quad\\Longleftrightarrow\\quad x_t = \\sqrt{\\bar\\alpha_t}\\,x_0 + \\sqrt{1-\\bar\\alpha_t}\\,\\epsilon,\\ \\ \\epsilon\\sim\\mathcal{N}(0,I). $$Lemma: sum of independent Gaussians We will repeatedly use the following. If $u\\sim\\mathcal N(0,a^2 I)$ and $v\\sim\\mathcal N(0,b^2 I)$ are independent, then $u+v\\sim\\mathcal N(0,(a^2+b^2)I)$. This holds because the sum of independent Gaussians is Gaussian (closure under convolution), the mean is $0+0=0$ by linearity, and for independent variables variances add: $\\mathrm{Var}(u+v)=\\mathrm{Var}(u)+\\mathrm{Var}(v)=(a^2+b^2)I$, the cross term $2\\,\\mathrm{Cov}(u,v)$ vanishing by independence. Crucially, in the chain each $\\epsilon_{t-1}$ is drawn fresh and independently of all $\\epsilon_{\\lt t-1}$ and of $x_0$, so the independence hypothesis is satisfied at every collapse below.\nDerivation by induction We prove $q(x_t\\mid x_0)=\\mathcal N(\\sqrt{\\bar\\alpha_t}\\,x_0,(1-\\bar\\alpha_t)I)$ by induction on $t$.\nBase case ($t=1$). From the single step, $x_1=\\sqrt{\\alpha_1}\\,x_0+\\sqrt{1-\\alpha_1}\\,\\epsilon_0$. Since $\\bar\\alpha_1=\\alpha_1$, this is exactly $x_1=\\sqrt{\\bar\\alpha_1}\\,x_0+\\sqrt{1-\\bar\\alpha_1}\\,\\epsilon_0$, i.e. $q(x_1\\mid x_0)=\\mathcal N(\\sqrt{\\bar\\alpha_1}\\,x_0,(1-\\bar\\alpha_1)I)$. ✓\nInductive step. Assume $x_{t-1}=\\sqrt{\\bar\\alpha_{t-1}}\\,x_0+\\sqrt{1-\\bar\\alpha_{t-1}}\\,\\bar\\epsilon_{t-1}$ with $\\bar\\epsilon_{t-1}\\sim\\mathcal N(0,I)$ independent of the next-drawn $\\epsilon_{t-1}$. Apply one more forward step:\n$$ x_t = \\sqrt{\\alpha_t}\\,x_{t-1}+\\sqrt{1-\\alpha_t}\\,\\epsilon_{t-1} = \\sqrt{\\alpha_t}\\Big(\\sqrt{\\bar\\alpha_{t-1}}\\,x_0+\\sqrt{1-\\bar\\alpha_{t-1}}\\,\\bar\\epsilon_{t-1}\\Big)+\\sqrt{1-\\alpha_t}\\,\\epsilon_{t-1}. $$The deterministic part is $\\sqrt{\\alpha_t\\bar\\alpha_{t-1}}\\,x_0=\\sqrt{\\bar\\alpha_t}\\,x_0$, since $\\alpha_t\\bar\\alpha_{t-1}=\\bar\\alpha_t$. The two noise terms,\n$$ \\underbrace{\\sqrt{\\alpha_t(1-\\bar\\alpha_{t-1})}\\,\\bar\\epsilon_{t-1}}_{\\text{var }=\\,\\alpha_t(1-\\bar\\alpha_{t-1})} \\;+\\; \\underbrace{\\sqrt{1-\\alpha_t}\\,\\epsilon_{t-1}}_{\\text{var }=\\,1-\\alpha_t}, $$are independent zero-mean Gaussians, so by the Lemma their sum is $\\mathcal N(0,c\\,I)$ with\n$$ c = \\alpha_t(1-\\bar\\alpha_{t-1}) + (1-\\alpha_t) = \\alpha_t - \\alpha_t\\bar\\alpha_{t-1} + 1 - \\alpha_t = 1 - \\alpha_t\\bar\\alpha_{t-1} = 1 - \\bar\\alpha_t. $$Hence $x_t=\\sqrt{\\bar\\alpha_t}\\,x_0+\\sqrt{1-\\bar\\alpha_t}\\,\\bar\\epsilon_t$ with $\\bar\\epsilon_t\\sim\\mathcal N(0,I)$, i.e. $q(x_t\\mid x_0)=\\mathcal N(\\sqrt{\\bar\\alpha_t}\\,x_0,(1-\\bar\\alpha_t)I)$. This is the claim at level $t$, completing the induction. $\\blacksquare$\nRemark (one telescoping line). Unrolling the recursion to $x_0$ directly, the signal coefficient telescopes to $\\prod_{s=1}^t\\sqrt{\\alpha_s}=\\sqrt{\\bar\\alpha_t}$, and the accumulated independent noise contributes variance $\\sum_{s=1}^t \\big(\\prod_{r\\gt s}\\alpha_r\\big)(1-\\alpha_s)$, a telescoping sum equal to $1-\\bar\\alpha_t$. The induction above is just the bookkeeping for this collapse.\nWhy this is the most important equation in the course Cheap training. A noisy sample at level $t$ costs one operation: sample $\\epsilon$, compute $\\sqrt{\\bar\\alpha_t}\\,x_0+\\sqrt{1-\\bar\\alpha_t}\\,\\epsilon$. No sequential simulation. Signal–noise interpretation. $\\sqrt{\\bar\\alpha_t}$ is the surviving signal, $\\sqrt{1-\\bar\\alpha_t}$ the accumulated noise; their squares sum to $1$, so $x_t$ keeps unit variance. The label is the noise. Once $x_t=\\sqrt{\\bar\\alpha_t}\\,x_0+\\sqrt{1-\\bar\\alpha_t}\\,\\epsilon$, the specific $\\epsilon$ drawn is the ground truth the network will predict — the seed of the Lesson 5 loss. Figure 2. Seven hundred patient steps, or one.\nThe noise schedule and signal-to-noise ratio We have been treating the $\\beta_t$\u0026rsquo;s as simply given; it is time to ask who chooses them, and how that choice shapes the entire descent from photograph to static.\nDefine the signal-to-noise ratio at step $t$ as\n$$ \\mathrm{SNR}(t) := \\frac{\\bar\\alpha_t}{1-\\bar\\alpha_t}, \\qquad \\log\\mathrm{SNR}(t) = \\log\\bar\\alpha_t - \\log(1-\\bar\\alpha_t). $$$\\mathrm{SNR}(t)$ decreases monotonically from $\\mathrm{SNR}(0)=\\infty$ (clean data) toward $\\mathrm{SNR}(T)\\approx 0$ (pure noise). The schedule controls how fast $\\log\\mathrm{SNR}$ descends, and recent work shows the objective and sampling can be parameterized entirely through $\\log\\mathrm{SNR}$ rather than $\\beta_t$ (Kingma et al., 2021).\nLinear schedule (DDPM). $\\beta_t$ increases linearly, e.g. from $\\beta_1=10^{-4}$ to $\\beta_T=0.02$ over $T=1000$. Simple, but $\\log\\mathrm{SNR}$ collapses quickly near the end, so the last steps carry little information.\nCosine schedule (Nichol \u0026amp; Dhariwal). Defines $\\bar\\alpha_t$ directly to keep more information at high $t$:\n$$ \\bar\\alpha_t = \\frac{f(t)}{f(0)}, \\qquad f(t) = \\cos\\!\\left(\\frac{t/T + s}{1+s}\\cdot\\frac{\\pi}{2}\\right)^2, $$for a small offset $s$ (e.g. $0.008$), with $\\beta_t=1-\\bar\\alpha_t/\\bar\\alpha_{t-1}$ clipped below $\\sim0.999$ for stability. This yields a gentler, near-linear $\\log\\mathrm{SNR}$ decay and improves both likelihood and sample quality.\nRemark (continuous-time / VP-SDE). Letting $T\\to\\infty$ and $\\beta_t\\to\\beta(t)\\,dt$, the forward chain becomes the variance-preserving SDE $dx = -\\tfrac12\\beta(t)\\,x\\,dt + \\sqrt{\\beta(t)}\\,dw$ (Song et al., 2021). Then $\\bar\\alpha_t=\\exp\\!\\big(-\\!\\int_0^t\\beta(s)\\,ds\\big)$, and the discrete VP chain is its Euler–Maruyama discretization. The schedule is just the choice of $\\beta(t)$, equivalently the $\\log\\mathrm{SNR}$ trajectory.\nFigure 3. Where the two standard schedules spend their information.\nThe boundary conditions As $t\\to T$, the schedule is designed so $\\bar\\alpha_T\\approx 0$, giving $q(x_T\\mid x_0)\\approx\\mathcal N(0,I)$ — the data is washed out and the endpoint is (almost) a standard Gaussian, independent of $x_0$. This is essential: at sampling time we start from $\\mathcal N(0,I)$ without any $x_0$, and that launchpad must match $q(x_T)$. At the other end, $\\bar\\alpha_0=1$ (empty product), so $x_0$ is the clean data.\nCode \u0026amp; numerics A reference implementation of the schedule and the one-shot forward jump. Shapes assume images $x_0$ of shape (B, C, H, W).\nimport torch def make_alpha_bar(T=1000, beta_start=1e-4, beta_end=2e-2, device=\u0026#34;cpu\u0026#34;): # betas: (T,) -\u0026gt; alphas: (T,) -\u0026gt; alpha_bar via cumulative product betas = torch.linspace(beta_start, beta_end, T, device=device) # (T,) alphas = 1.0 - betas # (T,) alpha_bar = torch.cumprod(alphas, dim=0) # (T,) = prod_{s\u0026lt;=t} alpha_s return betas, alphas, alpha_bar def forward_jump(x0, t, alpha_bar): # x0: (B, C, H, W) ; t: (B,) integer step indices in [0, T) # returns x_t and the noise epsilon used as the training label ab_t = alpha_bar[t] # (B,) # reshape for broadcasting against (B, C, H, W) sqrt_ab = ab_t.sqrt().view(-1, 1, 1, 1) # (B,1,1,1) sqrt_1mab = (1.0 - ab_t).clamp(min=0).sqrt().view(-1, 1, 1, 1) eps = torch.randn_like(x0) # (B, C, H, W) ~ N(0, I) x_t = sqrt_ab * x0 + sqrt_1mab * eps # closed-form q(x_t | x_0) return x_t, eps Numerical notes.\nUse torch.cumprod once and cache alpha_bar; never recompute the product per batch. 1 - alpha_bar can underflow toward $0$ at small $t$ and toward machine-epsilon error near $t=0$; .clamp(min=0) guards the sqrt against tiny negative values from float rounding. Near $t=T$, $\\bar\\alpha_T$ is extremely small but should stay $\\gt 0$ — if a schedule drives it to exactly $0$, $\\mathrm{SNR}$ and any $1/\\sqrt{\\bar\\alpha_t}$ term (used in $x_0$-prediction reparameterizations) diverge. Prefer float32 (or float64 for the cumprod) when $T$ is large: a long product of numbers $\\lt 1$ accumulates rounding; computing $\\log\\bar\\alpha_t=\\sum_s\\log\\alpha_s$ via torch.cumsum(torch.log(alphas)) is more stable, then exponentiate. Index alpha_bar[t] with an integer tensor of shape (B,), then .view(-1,1,1,1) to broadcast cleanly across channels and spatial dims. Exercises 1. Variance preservation. Assume each coordinate of $x_{t-1}$ has unit variance and is independent of the fresh noise $\\epsilon_{t-1}\\sim\\mathcal N(0,I)$. Show the single forward step $x_t=\\sqrt{1-\\beta_t}\\,x_{t-1}+\\sqrt{\\beta_t}\\,\\epsilon_{t-1}$ leaves the variance equal to $1$.\n$$ \\mathrm{Var}(x_t)=(1-\\beta_t)\\,\\mathrm{Var}(x_{t-1})+\\beta_t\\,\\mathrm{Var}(\\epsilon_{t-1})=(1-\\beta_t)\\cdot1+\\beta_t\\cdot1=1, $$the cross term $2\\sqrt{(1-\\beta_t)\\beta_t}\\,\\mathrm{Cov}(x_{t-1},\\epsilon_{t-1})$ vanishing by independence. Hence \u0026ldquo;variance-preserving.\u0026rdquo;\n2. Moments and SNR. Using the closed form $x_t=\\sqrt{\\bar\\alpha_t}\\,x_0+\\sqrt{1-\\bar\\alpha_t}\\,\\epsilon$, compute $\\mathbb E[x_t\\mid x_0]$ and $\\mathrm{Var}(x_t\\mid x_0)$, and confirm that the ratio of signal power to noise power is $\\mathrm{SNR}(t)=\\bar\\alpha_t/(1-\\bar\\alpha_t)$ (take $\\mathrm{Var}(x_0)\\approx1$).\n$\\mathbb E[x_t\\mid x_0]=\\sqrt{\\bar\\alpha_t}\\,x_0$ (since $\\mathbb E[\\epsilon]=0$) and $\\mathrm{Var}(x_t\\mid x_0)=(1-\\bar\\alpha_t)I$. The deterministic \u0026ldquo;signal\u0026rdquo; part carries power $\\bar\\alpha_t\\,\\mathrm{Var}(x_0)\\approx\\bar\\alpha_t$; the noise part carries power $1-\\bar\\alpha_t$. Their ratio is $\\bar\\alpha_t/(1-\\bar\\alpha_t)=\\mathrm{SNR}(t)$.\n3. A two-step jump from scratch. Without invoking the general closed form, compose two forward steps to express $x_t$ in terms of $x_{t-2}$, and show $x_t=\\sqrt{\\alpha_t\\alpha_{t-1}}\\,x_{t-2}+\\sqrt{1-\\alpha_t\\alpha_{t-1}}\\,\\bar\\epsilon$ with $\\bar\\epsilon\\sim\\mathcal N(0,I)$. (This is the induction\u0026rsquo;s inductive step in miniature.)\nWrite $x_t=\\sqrt{\\alpha_t}\\,x_{t-1}+\\sqrt{1-\\alpha_t}\\,\\epsilon_{t-1}$ and $x_{t-1}=\\sqrt{\\alpha_{t-1}}\\,x_{t-2}+\\sqrt{1-\\alpha_{t-1}}\\,\\epsilon_{t-2}$. Substituting,\n$$ x_t=\\sqrt{\\alpha_t\\alpha_{t-1}}\\,x_{t-2}+\\underbrace{\\sqrt{\\alpha_t(1-\\alpha_{t-1})}\\,\\epsilon_{t-2}+\\sqrt{1-\\alpha_t}\\,\\epsilon_{t-1}}_{\\text{two independent zero-mean Gaussians}}. $$By the variance-addition lemma the noise term is $\\mathcal N(0,c\\,I)$ with $c=\\alpha_t(1-\\alpha_{t-1})+(1-\\alpha_t)=1-\\alpha_t\\alpha_{t-1}$. Hence $x_t=\\sqrt{\\alpha_t\\alpha_{t-1}}\\,x_{t-2}+\\sqrt{1-\\alpha_t\\alpha_{t-1}}\\,\\bar\\epsilon$ — the $\\bar\\alpha$ pattern with a two-factor product.\n4. Numerical stability of $\\bar\\alpha_t$. Explain why computing $\\log\\bar\\alpha_t=\\sum_{s\\le t}\\log\\alpha_s$ via a cumulative sum of logs is more stable for $T\\sim1000$ than forming $\\bar\\alpha_t=\\prod_{s\\le t}\\alpha_s$ and then taking a log.\n$\\bar\\alpha_t$ is a product of $\\sim\\!1000$ factors each in $(0,1)$, so it shrinks toward (and near $t=T$ very close to) zero — the running product underflows and accumulates multiplicative rounding error, and $\\log$ of a near-zero value is ill-conditioned. Summing logs keeps every summand $O(1)$, avoids underflow entirely, and accumulates only additive round-off; exponentiating at the end recovers $\\bar\\alpha_t$ accurately.\nConclusion The forward process is a fixed, variance-preserving Gaussian Markov chain that contracts the signal and injects noise step by step. Its defining gift is the closed-form jump $x_t=\\sqrt{\\bar\\alpha_t}\\,x_0+\\sqrt{1-\\bar\\alpha_t}\\,\\epsilon$ — which we proved rigorously by induction using the independence and variance-addition of Gaussians — letting us sample any noise level instantly and exposing $\\epsilon$ as the natural training target. We made the schedule precise (linear vs. cosine), defined $\\mathrm{SNR}=\\bar\\alpha_t/(1-\\bar\\alpha_t)$ and its continuous-time VP-SDE limit, and saw why $x_T$ lands on $\\mathcal N(0,I)$. In Lesson 3 we turn around: what does the reverse step look like, why is it also Gaussian, and what exactly must the network compute?\nThe teleport is in hand: we can now stand at any noise level instantly, the seven-hundredth speck reached in a single stroke. That settles the easy direction. In Lesson 3 we turn around to face the hard one and ask what it would take to step back toward the light — and why, surprisingly, the answer is again a Gaussian.\nReferences Ho, J., Jain, A., \u0026amp; Abbeel, P. (2020). Denoising Diffusion Probabilistic Models (DDPM). NeurIPS. Nichol, A., \u0026amp; Dhariwal, P. (2021). Improved Denoising Diffusion Probabilistic Models (cosine schedule). ICML. Kingma, D. P., Salimans, T., Poole, B., \u0026amp; Ho, J. (2021). Variational Diffusion Models (SNR / log-SNR view). NeurIPS. Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., \u0026amp; Poole, B. (2021). Score-Based Generative Modeling through Stochastic Differential Equations (VP-SDE). ICLR. ← Previous: Lesson 1 — The Big Picture · Course Index · Next: Lesson 3 — The Reverse Process → · Appendix A: Acronyms · Appendix B: Formulas The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/tutorials/diffusion-lessons/lesson-02-forward-process/","summary":"\u003ch1 id=\"lesson-2--the-forward-process-destroying-structure-with-noise\"\u003eLesson 2 — The Forward Process: Destroying Structure with Noise\u003c/h1\u003e\n\u003cblockquote\u003e\n\u003cp\u003e\u003cstrong\u003eRecap of Lesson 1.\u003c/strong\u003e A diffusion model is built on an asymmetry: a \u003cem\u003efixed\u003c/em\u003e forward process $q$ that gradually turns data $x_0$ into pure noise $x_T$, and a \u003cem\u003elearned\u003c/em\u003e reverse process $p_\\theta$ that denoises step by step. Generation = running the learned reverse from noise. We met the single forward step $q(x_t\\mid x_{t-1}) = \\mathcal{N}(x_t; \\sqrt{1-\\beta_t}\\,x_{t-1}, \\beta_t I)$ and the assumed-Gaussian reverse step. Now we make the forward side completely rigorous.\u003c/p\u003e","title":"The Forward Process"},{"content":"Lesson 2 — Tokenization \u0026amp; Embeddings Technical course on LLM mechanics — Lesson 2 of 10.\nRecap of Lesson 1 A model is a function $f(x;\\theta)$ from a token sequence to a categorical distribution over the next token, and the chain rule $P_\\theta(x_{1:n})=\\prod_t P_\\theta(x_t\\mid x_{\\lt t})$ makes that sufficient to model whole sequences. Training minimizes $\\mathcal L(\\theta)$ via a forward/backward/update loop and is the only phase that writes $\\theta$; inference evaluates $f$ at frozen $\\theta$. We separate three kinds of numbers — [LEARNED] parameters, [CONFIG] hyperparameters (which fix the shape of $\\theta$), and [STATE] transient values — and quantified cost as $\\approx 6PD$ FLOPs to train, $\\approx 2P$/token to infer.\nLesson 1 treated $f$ as a black box with a known input/output type. Now we start opening it, from the bottom. The very first thing $f$ needs is numbers to compute on, but what arrives is a string of human text. This lesson specifies the bottom of $f$: the map from a Unicode string to the input tensor $H^{(0)}\\in\\mathbb R^{n\\times d}$ — the first numerical representation everything above it will consume.\n1. The interface problem $f$ operates on $\\mathbb R^{n\\times d}$ tensors; the input is a byte string. The bridge is two maps composed: a non-differentiable, fixed tokenizer $\\tau:\\text{string}\\to\\mathbb Z_{\\ge 0}^{n}$, and a differentiable, learned embedding $\\mathcal E:\\mathbb Z_{\\ge0}^n\\to\\mathbb R^{n\\times d}$. The split matters: gradients flow through $\\mathcal E$ (it is part of $\\theta$) but stop at $\\tau$ (it is [CONFIG], frozen before training). This is the first concrete instance of the Lesson 1 distinction: $\\tau$ is chosen, $\\mathcal E$ is learned. We take the two maps in order — first $\\tau$, then $\\mathcal E$.\nFigure 1. The bridge from a byte string to a tensor, and where the gradient stops.\n2. Subword tokenization Why subwords, formally We need a finite vocabulary $\\mathcal V$, $|\\mathcal V|=V$, that achieves (i) closure — every possible input is representable — and (ii) short sequences — few tokens per unit of text, since attention cost is $O(n^2)$ in sequence length $n$ (Lesson 3). Word-level vocabularies fail closure (open vocabulary, OOV words); pure byte/char vocabularies satisfy closure but blow up $n$. Subword vocabularies interpolate: frequent strings become single tokens (small $n$), and closure is guaranteed by a byte-level fallback.\nByte-Pair Encoding (BPE) BPE learns the vocabulary by greedy frequency-based merging. Given a corpus pre-split into words represented as byte sequences:\nInitialize $\\mathcal V$ with all 256 byte values (guarantees closure). Count the frequency of every adjacent symbol pair across the corpus. Merge the most frequent pair $(a,b)\\to ab$, add the new symbol $ab$ to $\\mathcal V$, and record the ordered merge rule. Repeat until $|\\mathcal V|=V$ (a [CONFIG] target, typically $3\\!\\times\\!10^4$–$2\\!\\times\\!10^5$). At encode time the recorded merges are reapplied in learned order; at decode time tokens are concatenated and the bytes UTF-8-decoded. The output for a sequence is a vector of integer IDs:\n$$ \\texttt{\"The cat sat\"}\\ \\xrightarrow{\\ \\tau\\ }\\ (\\,464,\\ 3797,\\ 3332\\,)\\in\\mathbb Z^{3}. $$These IDs are nominal, not ordinal — ID 3797 is not \u0026ldquo;more\u0026rdquo; than 464. Arithmetic on IDs is meaningless, which forces the embedding step. (Variants: WordPiece, used by BERT, merges by likelihood gain rather than raw frequency; Unigram/SentencePiece prunes a large vocabulary by marginal likelihood. All produce a fixed, frozen $\\tau$.)\n[CONFIG] $\\tau$, $V$, and the merge table are fixed before training and never updated. Tokenization choices have real downstream effects — e.g. digits and whitespace handling affect arithmetic and code performance — but they are design decisions, not learned quantities.\nFigure 2. Closure or short sequences — and the compromise that shipped.\n3. The embedding map Tokenization left us with integer IDs that carry no usable geometry. The embedding map is where those nominal labels become vectors that can be added, compared, and — crucially — learned. This is the first piece of $\\theta$ we meet.\nLookup as matrix multiplication Define the learned embedding matrix\n$$ E\\in\\mathbb R^{V\\times d},\\qquad d=\\text{model width (}\\textbf{[CONFIG]}\\text{, e.g. }768,\\,4096,\\,8192). $$Embedding token ID $i$ is row selection, $\\mathbf x_i = E_{i,:}$. Formally this is a linear map applied to a one-hot vector $\\mathbf e_i\\in\\{0,1\\}^V$ (with $(\\mathbf e_i)_j=\\delta_{ij}$):\n$$ \\mathbf x_i = \\mathbf e_i^\\top E \\in\\mathbb R^{d}. $$Writing it as a matrix product is not pedantry: it tells us the gradient. Since $\\mathbf x_i=\\mathbf e_i^\\top E$, the Jacobian routes the upstream gradient $\\partial\\mathcal L/\\partial\\mathbf x_i$ to only the selected row:\n$$ \\frac{\\partial\\mathcal L}{\\partial E_{j,:}}=\\sum_{\\text{positions }i\\,:\\,x_i=j}\\frac{\\partial\\mathcal L}{\\partial \\mathbf x_i}. $$So in a given step, only embeddings of tokens that actually appeared receive a gradient — the embedding update is sparse in the rows of $E$. The parameter count of $E$ is $Vd$, which for large $V,d$ is a substantial slice of $P$ (and motivates weight tying, Lesson 5).\nThe induced geometry $E$ is initialized randomly (e.g. $\\mathcal N(0,\\sigma^2)$) and learned. Training discovers a geometry in which the next-token objective is easy, which empirically produces semantically structured embeddings: cosine similarity tracks distributional similarity, and certain linear directions encode relational structure (the canonical $E_{\\text{king}}-E_{\\text{man}}+E_{\\text{woman}}\\approx E_{\\text{queen}}$). This structure is emergent, a byproduct of compression for prediction, not an imposed constraint.\n4. Injecting position Embeddings give each token a meaning vector, but they say nothing about where the token sits. That omission is harmless for the embedding itself and fatal for the layer above it. First, the intuition for why this step is needed at all. Self-attention (Lesson 3) treats its input as a set, not a sequence: permuting the input rows permutes the output rows identically, so by itself it literally cannot tell \u0026ldquo;dog bites man\u0026rdquo; from \u0026ldquo;man bites dog.\u0026rdquo; Word order carries meaning, so we must hand the model position information explicitly. Three schemes dominate; the first two are easy to state, and the third — RoPE — is the modern default and gets the most space.\n(a) Sinusoidal (fixed). The original transformer adds a fixed signal to the embedding at each position $p$, built from sines and cosines of $p$ at geometrically spaced frequencies, $PE_{(p,2k)}=\\sin(p/10000^{2k/d})$ and $PE_{(p,2k+1)}=\\cos(p/10000^{2k/d})$. Each position gets a unique signature, and because $PE_{p+\\Delta}$ is a fixed rotation of $PE_p$, relative offsets are linearly recoverable. No parameters.\n(b) Learned absolute. Replace that fixed signal with a trainable table $P_{\\text{pos}}\\in\\mathbb R^{n_{\\max}\\times d}$ (one row per position), added the same way. Simple and effective in-distribution, but it has no row for positions beyond the trained length $n_{\\max}$, so it cannot extrapolate to longer contexts.\n(c) Rotary position embeddings — RoPE (now standard). Here is the key intuition: what attention actually needs is not where each token sits absolutely, but how far apart two tokens are. RoPE delivers exactly that. Instead of adding a position signal to the embedding, it rotates the query and key vectors (Lesson 3) by an angle proportional to their position, working on 2-D coordinate pairs. For pair $k$ at position $p$ with frequency $\\omega_k=10000^{-2k/d}$:\n$$ \\begin{pmatrix}q'_{2k}\\\\ q'_{2k+1}\\end{pmatrix} =\\begin{pmatrix}\\cos p\\omega_k \u0026 -\\sin p\\omega_k\\\\ \\sin p\\omega_k \u0026 \\cos p\\omega_k\\end{pmatrix} \\begin{pmatrix}q_{2k}\\\\ q_{2k+1}\\end{pmatrix}. $$Why this is the clever part: when a query at position $p$ is dotted with a key at position $m$, the two rotations combine, and the result depends on $p$ and $m$ only through the difference $p-m$. So a relative-position signal falls out of the ordinary attention dot product for free — no extra terms, no parameters. A token 5 places back is treated the same way whether the pair sits at positions 10–15 or 1010–1015, which is precisely the property absolute schemes lack. That translation-invariance is why RoPE generalizes across lengths and why nearly every modern context-extension trick (position interpolation, NTK-aware scaling, YaRN — see Appendix D) operates by rescaling these rotation frequencies. The frequencies are fixed config; RoPE adds no parameters.\nFigure 3. Why RoPE gives a relative signal from an absolute rotation.\n5. The input tensor Tokenizer, embedding, and position now assemble into the single object the rest of the network reads. Combining the pieces, the input to the transformer stack for an $n$-token sequence is\n$$ H^{(0)}=\\mathcal E(\\tau(\\text{text})) \\ (+\\,PE\\ \\text{if additive scheme})\\ \\in\\mathbb R^{n\\times d}, $$one $d$-vector per position. The superscript indexes layer depth: $H^{(0)}$ is pre-stack, and block $\\ell$ maps $H^{(\\ell-1)}\\!\\to\\! H^{(\\ell)}$ (Lesson 4).\nLearned / config / state in this lesson Quantity Category Tokenizer $\\tau$, vocab $V$, merge table [CONFIG] Width $d$, max context $n_{\\max}$ [CONFIG] Embedding matrix $E\\in\\mathbb R^{V\\times d}$ [LEARNED] Learned positional table $P_{\\text{pos}}$ (if used) [LEARNED] Sinusoidal / RoPE frequencies [CONFIG] (no params) Token IDs, $H^{(0)}$ [STATE] (per-input activations) Conclusion Text reaches $f$ through a fixed, non-differentiable tokenizer $\\tau$ and a learned embedding $E\\in\\mathbb R^{V\\times d}$. BPE learns $\\tau$ by greedy frequency merging from a byte-level base, guaranteeing closure while keeping sequences short — important because attention is $O(n^2)$. Embedding is a one-hot–selected linear map $\\mathbf x_i=\\mathbf e_i^\\top E$, which makes the row-sparse gradient explicit and accounts for $Vd$ of the parameter budget; the resulting geometry is emergent. Because attention is permutation-equivariant, position is injected separately — via fixed sinusoidal encodings, learned absolute tables, or RoPE, which encodes relative position inside the attention dot product and is now the default. The output $H^{(0)}\\in\\mathbb R^{n\\times d}$ is the first numerical representation the rest of the network computes on.\nBut each row of $H^{(0)}$ was computed in isolation: the vector for a pronoun knows nothing yet about the noun it refers to. Next we derive the mechanism that lets these per-token vectors exchange information — the first place the sequence starts to behave like more than a bag of independent tokens.\n← Lesson 1 — The Big Picture · Course index · Next: Lesson 3 — Self-Attention → Appendices: A — Acronyms · B — Formulas · C — Attention Variants The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/tutorials/llm-lessons/lesson-02-tokenization-and-embeddings/","summary":"\u003ch1 id=\"lesson-2--tokenization--embeddings\"\u003eLesson 2 — Tokenization \u0026amp; Embeddings\u003c/h1\u003e\n\u003cp\u003e\u003cem\u003eTechnical course on LLM mechanics — Lesson 2 of 10.\u003c/em\u003e\u003c/p\u003e\n\u003chr\u003e\n\u003ch2 id=\"recap-of-lesson-1\"\u003eRecap of Lesson 1\u003c/h2\u003e\n\u003cp\u003eA model is a function $f(x;\\theta)$ from a token sequence to a categorical distribution over the next token, and the chain rule $P_\\theta(x_{1:n})=\\prod_t P_\\theta(x_t\\mid x_{\\lt t})$ makes that sufficient to model whole sequences. Training minimizes $\\mathcal L(\\theta)$ via a forward/backward/update loop and is the only phase that writes $\\theta$; inference evaluates $f$ at frozen $\\theta$. We separate three kinds of numbers — \u003cstrong\u003e[LEARNED]\u003c/strong\u003e parameters, \u003cstrong\u003e[CONFIG]\u003c/strong\u003e hyperparameters (which fix the \u003cem\u003eshape\u003c/em\u003e of $\\theta$), and \u003cstrong\u003e[STATE]\u003c/strong\u003e transient values — and quantified cost as $\\approx 6PD$ FLOPs to train, $\\approx 2P$/token to infer.\u003c/p\u003e","title":"Tokenization and Embeddings"},{"content":"Lesson 2 — Tokenization \u0026amp; the Encoder Input Technical course on encoder LLM mechanics — Lesson 2 of 10.\nRecap of Lesson 1 An encoder is a function $g(x;\\theta):\\mathcal V^{n}\\to\\mathbb R^{n\\times d}$ that maps a token sequence to a matrix of contextual representations, each row informed by the whole input bidirectionally. It has no autoregressive factorization, so it is trained by masked reconstruction rather than next-token prediction (Lesson 5), it produces representations rather than a next-token distribution (so it needs task heads, Lesson 6), and it runs in a single parallel forward pass with no KV cache. We keep the [LEARNED]/[CONFIG]/[STATE] tagging throughout.\nLesson 1 treated $g$ as a black box with a known output type. Now we open it from the bottom. The very first thing $g$ needs is numbers, but what arrives is a string. This lesson builds the encoder\u0026rsquo;s input tensor $H^{(0)}\\in\\mathbb R^{n\\times d}$ — and it turns out an encoder constructs that tensor from three summed pieces, not the decoder\u0026rsquo;s one, and frames every input with special marker tokens that have no decoder analogue.\n1. The interface problem (shared) and what changes As in the decoder, the bridge from string to tensor is two composed maps: a fixed, non-differentiable tokenizer $\\tau:\\text{string}\\to\\mathbb Z_{\\ge0}^{n}$ and a learned, differentiable embedding $\\mathcal E:\\mathbb Z_{\\ge0}^{n}\\to\\mathbb R^{n\\times d}$. Gradients flow through $\\mathcal E$ (it is part of $\\theta$) and stop at $\\tau$ (it is [CONFIG], frozen before training). That structure is identical to the decoder.\nTwo things change for the encoder, and both come from §1 of Lesson 1. First, because the input is a complete, known sequence rather than a growing generation, the encoder wraps it in special tokens that mark its boundaries and provide a place to attach sequence-level predictions. Second, because BERT-style pretraining feeds the model pairs of segments, the embedding sum gains a segment component. We take the tokenizer first, then the three-part embedding.\n2. Subword tokenization: WordPiece The encoder\u0026rsquo;s goals for a vocabulary are the same as the decoder\u0026rsquo;s: a finite $\\mathcal V$, $|\\mathcal V|=V$, achieving closure (every input representable) and short sequences (few tokens per unit text, since attention is $O(n^2)$ in length). BERT uses WordPiece, a close cousin of the decoder\u0026rsquo;s BPE.\nBPE merges the most frequent adjacent pair at each step. WordPiece instead merges the pair that most increases the likelihood of the training corpus under a unigram language model — equivalently, it merges the pair $(a,b)$ maximizing\n$$ \\text{score}(a,b)=\\frac{\\mathrm{count}(ab)}{\\mathrm{count}(a)\\,\\mathrm{count}(b)}, $$so a pair is merged only when the bigram occurs more often than its parts would predict independently. The practical effect is similar to BPE — frequent strings become single tokens — but WordPiece tends to keep morphologically meaningful pieces together.\nA visible quirk: WordPiece marks continuation subwords with a ## prefix. The word \u0026ldquo;playing\u0026rdquo; might tokenize as play, ##ing; \u0026ldquo;embeddings\u0026rdquo; as em, ##bed, ##ding, ##s. The ## says \u0026ldquo;this piece attaches to the previous one with no space,\u0026rdquo; which lets the decoder side reconstruct spacing unambiguously.\n$$ \\texttt{\"playing games\"}\\ \\xrightarrow{\\ \\tau\\ }\\ (\\underbrace{2652}_{\\text{play}},\\ \\underbrace{2128}_{\\text{\\#\\#ing}},\\ \\underbrace{2399}_{\\text{games}})\\in\\mathbb Z^{3}. $$As with BPE, IDs are nominal (ID 2652 is not \u0026ldquo;more\u0026rdquo; than 2128), which forces the embedding step. BERT\u0026rsquo;s original vocabulary is $V=30{,}522$ [CONFIG] WordPiece tokens.\n[CONFIG] $\\tau$, $V$, the merge table, and the special-token IDs are all fixed before training and never updated.\nFigure 1. BPE and WordPiece: one goal, a different merge rule.\n3. Special tokens: the encoder\u0026rsquo;s framing This is the first piece with no decoder analogue. Every encoder input is wrapped in reserved symbols that the tokenizer inserts and the model learns to interpret.\n[CLS] — prepended to every input as position 0. Its job is to be a place to accumulate a whole-sequence summary: after the bidirectional stack runs, the output vector at the [CLS] position is used as the pooled representation for sequence-level tasks (Lesson 6). It is not a no-op; it is a learned \u0026ldquo;read-out register.\u0026rdquo; [SEP] — a separator that marks the end of a segment. A single-sentence input looks like [CLS] w₁ … w_k [SEP]. A sentence-pair input (for entailment, QA, etc.) looks like [CLS] a₁ … a_m [SEP] b₁ … b_n [SEP]. [MASK] — the corruption symbol used during MLM pretraining (Lesson 5). A masked position is replaced by this token, and the model is trained to recover the original. Crucially, [MASK] appears at pretraining time but never at fine-tuning or inference time — a train/test mismatch we will analyze in Lesson 5. [PAD] — pads a batch of variable-length sequences to a common length; padded positions are ignored via a mask (§5 of Lesson 3). [UNK] covers anything the tokenizer cannot represent (rare, given byte/character fallback). flowchart LR A[\"Question tokens\"] --\u003e C[\"[CLS] q₁ … q_m [SEP] p₁ … p_n [SEP]\"] B[\"Passage tokens\"] --\u003e C C --\u003e D[\"position 0 = [CLS]used for pooling\"] Contrast with the decoder. A decoder prompt is usually just the raw token stream (special tokens, if any, mark chat turns). An encoder always frames its input with [CLS]/[SEP], because it needs (a) a fixed slot to hang a sequence-level prediction on and (b) a way to express the sentence-pair inputs its tasks require. These tokens are part of the architecture\u0026rsquo;s contract, not optional decoration.\nFigure 2. How an encoder frames its input.\n4. The embedding map: three components summed Here is the second structural difference. The decoder\u0026rsquo;s input vector for a position is (token embedding) + (position information). BERT\u0026rsquo;s input vector is the sum of three learned embeddings:\n$$ \\boxed{\\ \\mathbf h^{(0)}_i=\\underbrace{E_{\\text{tok}}[x_i]}_{\\text{what token}}\\;+\\;\\underbrace{E_{\\text{seg}}[s_i]}_{\\text{which segment}}\\;+\\;\\underbrace{E_{\\text{pos}}[i]}_{\\text{which position}}\\ \\in\\mathbb R^{d}.\\ } $$All three are rows selected from learned matrices (the same one-hot-times-matrix lookup as the decoder\u0026rsquo;s token embedding), and all three are [LEARNED].\nToken embedding $E_{\\text{tok}}\\in\\mathbb R^{V\\times d}$. Standard: row $x_i$ is the embedding of token ID $x_i$. This is $Vd$ parameters with a row-sparse gradient (only embedded tokens get updated on a given step), exactly as in the decoder.\nSegment embedding $E_{\\text{seg}}\\in\\mathbb R^{2\\times d}$. The encoder-specific piece. Each token carries a segment id $s_i\\in\\{0,1\\}$: tokens of the first sentence get $s_i=0$, tokens of the second (after the first [SEP]) get $s_i=1$. The model thus knows, additively at the input, which half of a pair a token belongs to. Only two rows exist because BERT pairs at most two segments. (Single-sentence inputs use segment 0 throughout.)\nPosition embedding $E_{\\text{pos}}\\in\\mathbb R^{n_{\\max}\\times d}$. BERT uses learned absolute positions: position $i$ indexes a learned row, with $n_{\\max}=512$ [CONFIG] rows. This contrasts with the decoder course\u0026rsquo;s emphasis on RoPE (rotary, relative positions injected inside the attention dot-product). We pause on why the choice differs.\nWhy learned-absolute, and the 512 ceiling Sinusoidal and learned-absolute positions were the original Transformer/BERT choice; RoPE and ALiBi came later and now dominate decoders. Learned-absolute positions are simple and effective for the short, fixed-length inputs encoders typically handle (a sentence, a sentence pair, a paragraph). But they impose a hard length ceiling: there are exactly $n_{\\max}=512$ position rows, and the model has never seen position 512 or beyond, so it cannot meaningfully process longer inputs without retraining or interpolation. Decoders, which must generate arbitrarily long sequences, benefit far more from relative schemes that extrapolate. This is a real, practical encoder limitation — and exactly the thing ModernBERT (Lesson 10, Appendix A) fixes by switching to RoPE and extending context to 8192.\nContrast with the decoder. The decoder course\u0026rsquo;s position story climaxes with RoPE because generation needs length extrapolation. Classic encoders use learned-absolute positions with a fixed 512 ceiling, which is sufficient for sentence/paragraph understanding but caps input length. The summed segment embedding, meanwhile, has no decoder analogue at all.\nFigure 3. Three learned embeddings, summed into one row.\n5. Putting the input tensor together Stack the per-token vectors into rows and you have the encoder\u0026rsquo;s input:\n$$ H^{(0)}=\\big[\\mathbf h^{(0)}_1;\\ \\mathbf h^{(0)}_2;\\ \\dots;\\ \\mathbf h^{(0)}_n\\big]\\in\\mathbb R^{n\\times d}, $$typically followed by LayerNorm and dropout before the first transformer block. Every row already encodes what token it is, which segment it belongs to, and where it sits — but it knows nothing yet about the other tokens. Making the rows talk to each other is the job of bidirectional self-attention, Lesson 3.\nA concrete BERT-base configuration to anchor the shapes: $V=30{,}522$, $d=768$, $n_{\\max}=512$. The embedding parameter count is $\\underbrace{30{,}522\\cdot768}_{\\approx23.4\\text{M (token)}}+\\underbrace{2\\cdot768}_{\\text{segment}}+\\underbrace{512\\cdot768}_{\\approx0.39\\text{M (position)}}$ — token embeddings alone are over 20% of BERT-base\u0026rsquo;s ~110M parameters, a much larger fraction than in a wide decoder because $d$ is small.\nConclusion An encoder turns a string into $H^{(0)}\\in\\mathbb R^{n\\times d}$ through a fixed WordPiece tokenizer $\\tau$ ([CONFIG], likelihood-based merges, ## continuation marks) and a learned embedding that is the sum of three components: token, segment, and learned-absolute position — the latter capped at $n_{\\max}=512$. The input is framed by special tokens with no decoder analogue: [CLS] (a read-out register for sequence-level tasks), [SEP] (segment boundaries, enabling sentence-pair inputs), [MASK] (the pretraining corruption symbol), and [PAD]. Two of these choices — the segment embedding and the framing tokens — exist specifically because an encoder consumes complete, known, often paired inputs and must expose a slot for whole-sequence predictions.\nWith the input tensor built, Lesson 3 derives the operator that finally lets the rows of $H^{(0)}$ exchange information — and shows that removing the decoder\u0026rsquo;s causal mask is the single change that makes a model an encoder.\n← Previous: Lesson 1 — The Big Picture · Course index · Next: Lesson 3 — Bidirectional Self-Attention → Appendices: A — Encoder Family · B — Encoder vs. Decoder Cheat Sheet · C — Acronyms The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/tutorials/encoder-deep-dive/lesson-02-tokenization-and-embeddings/","summary":"\u003ch1 id=\"lesson-2--tokenization--the-encoder-input\"\u003eLesson 2 — Tokenization \u0026amp; the Encoder Input\u003c/h1\u003e\n\u003cp\u003e\u003cem\u003eTechnical course on encoder LLM mechanics — Lesson 2 of 10.\u003c/em\u003e\u003c/p\u003e\n\u003chr\u003e\n\u003ch2 id=\"recap-of-lesson-1\"\u003eRecap of Lesson 1\u003c/h2\u003e\n\u003cp\u003eAn encoder is a function $g(x;\\theta):\\mathcal V^{n}\\to\\mathbb R^{n\\times d}$ that maps a token sequence to a matrix of \u003cstrong\u003econtextual representations\u003c/strong\u003e, each row informed by the whole input bidirectionally. It has no autoregressive factorization, so it is trained by masked reconstruction rather than next-token prediction (Lesson 5), it produces representations rather than a next-token distribution (so it needs task heads, Lesson 6), and it runs in a single parallel forward pass with no KV cache. We keep the \u003cstrong\u003e[LEARNED]/[CONFIG]/[STATE]\u003c/strong\u003e tagging throughout.\u003c/p\u003e","title":"Tokenization and the Encoder Input"},{"content":"Chapter 3 — Attention: The Idea That Changed Everything At the end of the last chapter we left every word stranded on its own island. Each one had been placed on the map of meaning and stamped with its position, but each one\u0026rsquo;s meaning was still computed in isolation. The word \u0026ldquo;it\u0026rdquo; had no idea what it stood for. The word \u0026ldquo;bank\u0026rdquo; could not tell a river from a vault. To understand anything, the words must look at one another and revise themselves accordingly. The mechanism that lets them do this is called attention, and it is the breakthrough that made the modern era of AI possible. Almost every system in this book is built on it. So it is worth slowing down and getting the intuition exactly right.\nThe problem attention solves Consider the sentence: \u0026ldquo;The trophy didn\u0026rsquo;t fit in the suitcase because it was too big.\u0026rdquo; What does \u0026ldquo;it\u0026rdquo; refer to — the trophy or the suitcase? You know instantly that it is the trophy, because trophies that are too big are the reason things don\u0026rsquo;t fit. Now change one word: \u0026quot;\u0026hellip;because it was too small.\u0026quot; Suddenly \u0026ldquo;it\u0026rdquo; is the suitcase. The meaning of \u0026ldquo;it\u0026rdquo; depends entirely on the other words around it, and on which of them turn out to be relevant.\nThis is the everyday situation of language. The meaning of a word is rarely self-contained; it is shaped by particular other words, sometimes far away. \u0026ldquo;The man who I met last Tuesday in Paris — he said\u0026hellip;\u0026rdquo; The \u0026ldquo;he\u0026rdquo; reaches all the way back across the sentence to find \u0026ldquo;the man.\u0026rdquo; A model that processed each word in isolation would be lost. What we need is a mechanism by which each word can survey the whole sentence, decide which other words matter to it, and pull in information from exactly those words.\nThat is attention, in one sentence: a way for every word to look across the entire sentence, weigh how relevant each other word is to it, and update itself by drawing in the most relevant ones.\nFigure 1. One word changes, and \u0026ldquo;it\u0026rdquo; points somewhere else entirely.\nThe party analogy Here is a way to picture how attention actually decides what is relevant. Imagine the words of a sentence as people at a party, and imagine that each person carries three things.\nEach person carries a question — a statement of what they are looking for. The word \u0026ldquo;it,\u0026rdquo; for instance, is looking for whatever noun it refers to.\nEach person also wears a label — a short description of what they themselves are about. \u0026ldquo;Suitcase\u0026rdquo; wears a label saying, in effect, \u0026ldquo;I\u0026rsquo;m a container, a physical object, a thing that holds stuff.\u0026rdquo;\nAnd each person carries some information — the actual content they can contribute if someone finds them relevant.\nNow the mechanism runs. Each word takes its question and compares it against everyone else\u0026rsquo;s label, looking for strong matches. When \u0026ldquo;it\u0026rdquo; holds up its question and scans the room, the label on \u0026ldquo;suitcase\u0026rdquo; (or \u0026ldquo;trophy,\u0026rdquo; depending on the sentence) lights up as the best match. Having found its match, \u0026ldquo;it\u0026rdquo; pulls in the information that word carries, weighted by how strong the match was. Weak matches contribute little; strong matches contribute a lot. After this exchange, \u0026ldquo;it\u0026rdquo; is no longer a meaningless pronoun — it has absorbed the meaning of the thing it refers to. The islands have become connected. Every word, in parallel, does this same survey at the same time, so the whole sentence updates itself in one coordinated step.\nThis question-label-information dance is the entire idea. The technical names for the three things are query, key, and value, and you will meet them again, but the party picture is all you need to carry forward.\nFigure 2. The party: a question, a label, and some information.\nWhy this was such a breakthrough Before attention, the dominant approach processed text the way you might read with a finger on the page — strictly in order, carrying a running summary in your head, with older words fading as newer ones arrived. That made long-range connections fragile. By the time such a system reached \u0026ldquo;he\u0026rdquo; at the end of a long sentence, its memory of \u0026ldquo;the man\u0026rdquo; at the beginning had grown dim. These models struggled to hold a thought across distance.\nAttention threw out the running summary and let every word reach directly to any other word, no matter how far away, in a single step. Distance stopped being an obstacle. A word at the end of a paragraph can attend to a word at the very beginning just as easily as to its immediate neighbor. This direct, any-to-any reach is what let models finally handle long, intricate, deeply nested language — and it is why the architecture built around attention, called the Transformer, displaced everything that came before it and now underlies essentially every system in this book.\nMany conversations at once There is a refinement that matters. A single round of attention can really only track one kind of relationship at a time. But a sentence has many relationships running through it simultaneously: who refers to whom, which adjective modifies which noun, what the overall topic is, what the emotional tone is. One survey of the room cannot capture all of that at once.\nSo the model runs many attention processes side by side, in parallel — picture not one party but several happening in the same room, each attended to with a different question in mind. One of these parallel processes might come to specialize in tracking grammatical agreement, another in matching pronouns to their nouns, another in following the topic. Each forms its own answers, and the model then combines them. Critically, nobody assigns these specialties in advance. They emerge on their own during training, simply because dividing the labor this way helps the model predict better. (These parallel processes are called attention heads, and you will see the count of them — \u0026ldquo;twelve heads,\u0026rdquo; \u0026ldquo;ninety-six heads\u0026rdquo; — quoted as a basic spec of a model.)\nThe hidden price Attention\u0026rsquo;s great strength — that every word can look at every other word — is also the source of its central cost, and that cost shapes much of the engineering you will read about later. If every word must compare its question against every other word\u0026rsquo;s label, then the amount of work grows out of proportion to the length of the text. Double the length of a passage and you do not double the work; you roughly quadruple it, because every one of the now-twice-as-many words has to consider every other.\nThis is why processing long documents is expensive, why models have a \u0026ldquo;context window\u0026rdquo; beyond which they cannot see, and why a great deal of clever effort goes into making attention cheaper for long texts. Hold onto this fact. When later chapters talk about the cost of long conversations, or about tricks to handle book-length inputs, they are all wrestling with this same quadratic price tag that attention\u0026rsquo;s any-to-any reach demands.\nFigure 3. Double the length of a passage and the work roughly quadruples.\nWhat to carry forward Attention is the mechanism by which words stop being islands. Each word poses a question, every word offers a label and some information, and each word updates itself by drawing in the information of the words whose labels best answer its question. Run many such conversations in parallel and you can track many kinds of relationship at once. This single idea — direct, any-to-any, learned relevance — is what made modern AI work, and it is the beating heart of every architecture in Part II. Its only real drawback is that it grows expensive as text grows long, a tension we will return to repeatedly.\nWe now have the three foundations: what a model is (Chapter 1), how words become numbers on a map of meaning (Chapter 2), and how those numbers learn to talk to each other (Chapter 3). With attention in hand, we are ready to meet the family of language models — four different ways of arranging this same machinery, each built for a different purpose. We begin with the most famous of them: the storytellers. Turn to Chapter 4 .\n← Chapter 2 · From Words to Numbers · Contents · Chapter 4 · The Storytellers → The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/tutorials/inside-the-machines/chapter-03-attention/","summary":"\u003ch1 id=\"chapter-3--attention-the-idea-that-changed-everything\"\u003eChapter 3 — Attention: The Idea That Changed Everything\u003c/h1\u003e\n\u003cp\u003eAt the end of the last chapter we left every word stranded on its own island. Each one had been placed on the map of meaning and stamped with its position, but each one\u0026rsquo;s meaning was still computed in isolation. The word \u0026ldquo;it\u0026rdquo; had no idea what it stood for. The word \u0026ldquo;bank\u0026rdquo; could not tell a river from a vault. To understand anything, the words must look at one another and revise themselves accordingly. The mechanism that lets them do this is called \u003cem\u003eattention\u003c/em\u003e, and it is the breakthrough that made the modern era of AI possible. Almost every system in this book is built on it. So it is worth slowing down and getting the intuition exactly right.\u003c/p\u003e","title":"Attention: The Idea That Changed Everything"},{"content":"Lesson 3 — Bidirectional Self-Attention Technical course on encoder LLM mechanics — Lesson 3 of 10.\nRecap of Lesson 2 The encoder\u0026rsquo;s input tensor $H^{(0)}\\in\\mathbb R^{n\\times d}$ is the sum of three learned embeddings — token (WordPiece, $V=30{,}522$), segment ($s_i\\in\\{0,1\\}$), and learned-absolute position (capped at $n_{\\max}=512$) — with the sequence framed by [CLS], [SEP], [MASK], and [PAD]. Each row knows what token it is and where it sits, but nothing about the other tokens. This lesson derives the operator that fixes that, and shows that the only thing distinguishing encoder attention from decoder attention is the removal of one mask.\n1. The operator is the same; the mask is not Self-attention itself is identical to the decoder\u0026rsquo;s, so we restate it quickly and spend our effort on the one difference. Let $H\\in\\mathbb R^{n\\times d}$ be the input (rows = tokens). A head has three learned projections $W_Q,W_K\\in\\mathbb R^{d\\times d_k}$, $W_V\\in\\mathbb R^{d\\times d_v}$ producing\n$$ Q=HW_Q,\\quad K=HW_K,\\quad V=HW_V, $$and scaled dot-product attention is\n$$ \\mathrm{Attn}(Q,K,V)=\\underbrace{\\mathrm{softmax}\\!\\Big(\\tfrac{QK^\\top}{\\sqrt{d_k}}+M\\Big)}_{A\\in\\mathbb R^{n\\times n}}V, $$with the softmax applied row-wise. The $\\sqrt{d_k}$ scaling (which keeps the pre-softmax variance $O(1)$), the softmax Jacobian, and multi-head concatenation are exactly as in the decoder course; if they are rusty, revisit it. Everything in this lesson lives in the term $M$ — the mask.\n2. The causal mask, and deleting it In a decoder, $M$ is the causal mask: an $n\\times n$ matrix with $0$ on and below the diagonal and $-\\infty$ above it,\n$$ M^{\\text{causal}}_{ij}=\\begin{cases}0 \u0026 j\\le i\\\\ -\\infty \u0026 j\\gt i.\\end{cases} $$Adding $-\\infty$ to a score sends $\\exp(-\\infty)=0$ through the softmax, so position $i$ assigns exactly zero weight to any future position $j\\gt i$. That is what enforces the autoregressive factorization: the prediction at $i$ cannot see $x_{\\gt i}$.\nAn encoder simply removes this mask. For self-attention over the real (non-padding) tokens,\n$$ \\boxed{\\ M^{\\text{encoder}}_{ij}=0\\quad\\text{for all }i,j,\\ } $$so every position attends to every position, and the attention matrix $A$ is full (dense $n\\times n$), not lower-triangular:\n$$ \\mathbf o_i=\\sum_{j=1}^{n}A_{ij}\\,\\mathbf v_j,\\qquad A_{ij}=\\frac{\\exp(s_{ij})}{\\sum_{m=1}^{n}\\exp(s_{im})},\\qquad s_{ij}=\\frac{\\mathbf q_i^\\top\\mathbf k_j}{\\sqrt{d_k}}. $$The sum runs over all $n$ positions for every $i$, including $j\\gt i$. That single change — one matrix of zeros instead of a triangular matrix of $-\\infty$ — is the entire architectural difference between an encoder block and a decoder block. Token \u0026ldquo;it\u0026rdquo; at position 3 can now pull information from \u0026ldquo;dog\u0026rdquo; at position 6; the representation of every token is a function of the whole sentence.\nflowchart LR subgraph C[\"Causal mask (decoder): A is lower-triangular\"] c1[\"row i attends to j ≤ i only\"] end subgraph B[\"No mask (encoder): A is full\"] b1[\"row i attends to all j\"] end Contrast with the decoder. It is genuinely this simple at the operator level: encoder attention is decoder attention with the causal mask set to all-zeros. Every downstream difference — the objective, the absence of a KV cache, the inability to generate — is a consequence of this one change, not a separate design.\nFigure 1. Delete one mask and the whole matrix fills in.\n3. Why bidirectionality is incompatible with next-token training It is worth making the incompatibility from Lesson 1 rigorous, because it is the reason Lesson 5 exists. Suppose we tried to train a bidirectional model on the next-token loss $-\\sum_t\\log P_\\theta(x_t\\mid x_{\\lt t})$. The label for position $t$ is $x_t$ — but with no causal mask, the representation used to predict position $t$ has attended to position $t$ (and beyond), so $x_t$ is in scope as an input. The model can drive the loss to zero by learning the identity \u0026ldquo;copy the token you can already see.\u0026rdquo; The task carries no learning signal.\nFormally: next-token prediction is only a meaningful objective under the constraint that the predictor of $x_t$ is a function of $x_{\\lt t}$ alone. Bidirectional attention violates that constraint by construction. So bidirectionality and next-token training are mutually exclusive — you keep the causal mask (decoder) or you change the objective (encoder). BERT keeps bidirectionality and changes the objective; that is the trade at the heart of the family.\nInvariant. Bidirectional context buys richer representations at the cost of the next-token objective. There is no free lunch: you cannot have full context and train by predicting the next token.\nFigure 2. Why a bidirectional model cannot be trained on the next token.\n4. The cost, and what is missing relative to the decoder The compute cost of a bidirectional layer is the same order as a causal one: forming $QK^\\top$ is $O(n^2 d_k)$ and applying $AV$ is $O(n^2 d_v)$, so attention is $O(n^2 d)$ per layer. The encoder fills the entire $n\\times n$ matrix while a causal decoder fills only the lower triangle (about half), so an encoder does roughly $2\\times$ the attention arithmetic of a causal model at the same $n$ — a constant factor, not a different scaling.\nThe more important difference is what an encoder does not have:\nNo KV cache. The decoder caches each position\u0026rsquo;s keys and values so that generating token $t+1$ reuses the work done for $1,\\dots,t$. That optimization exists only because generation is an incremental left-to-right loop. An encoder processes the whole sequence in one shot and is then done; there is no \u0026ldquo;next step\u0026rdquo; to cache for. Consequently the encoder\u0026rsquo;s transient [STATE] at inference is just the attention matrices and activations of a single pass — no growing cache, no per-token memory blow-up.\nNo incremental decoding, hence no generation. Because every representation depends on the whole input, you cannot \u0026ldquo;extend\u0026rdquo; an encoder\u0026rsquo;s output one token at a time the way you sample from a decoder. There is no distribution over a next token to sample from. An encoder is a fixed-input → fixed-output map; turning it into a generator requires bolting a decoder onto it (encoder–decoder models, Lesson 10).\nContrast with the decoder. The decoder course spends a whole lesson (Inference in Depth) on the KV cache, GQA/MQA, and speculative decoding — all techniques for making the autoregressive loop affordable. None of them apply to an encoder, which has no loop. The encoder\u0026rsquo;s inference story is a single sentence: \u0026ldquo;run the stack once.\u0026rdquo;\nFigure 3. Twice the arithmetic, the same scaling.\n5. The mask that does survive: padding Encoders drop the causal mask but keep one mask of their own. To process a batch of variable-length sequences efficiently, short sequences are padded to a common length with [PAD] tokens (Lesson 2). We must stop real tokens from attending to padding, so we add a padding mask: for each query $i$, set $M_{ij}=-\\infty$ wherever $j$ is a padding position. This is content-driven (it depends on where the padding is), not structural like the causal mask, and it applies to columns (keys) uniformly across all query rows:\n$$ M^{\\text{pad}}_{ij}=\\begin{cases}0 \u0026 \\text{token }j\\text{ is real}\\\\ -\\infty \u0026 \\text{token }j\\text{ is }\\texttt{[PAD]}.\\end{cases} $$So the precise statement is: an encoder uses no causal mask but a padding mask; a decoder uses both a causal mask and (when batched) a padding mask. The padding mask carries no linguistic meaning — it is pure bookkeeping for batching.\nConclusion Bidirectional self-attention is scaled dot-product attention with the causal mask deleted: $M_{ij}=0$ for all real positions, so the attention matrix $A$ is full and every token\u0026rsquo;s representation is a function of the entire sequence, future included. This one change is what makes a model an encoder, and it has cascading consequences: it is incompatible with next-token training (the label is visible as input), which is why encoders need the masked objective of Lesson 5; it leaves the model with no KV cache and no ability to generate, since there is no incremental loop; and it costs roughly $2\\times$ the attention arithmetic of a causal model (filling the whole matrix rather than a triangle) at the same length. The only mask an encoder keeps is the padding mask, a batching detail with no linguistic content.\nWe now have context-aware token vectors. Lesson 4 stacks the attention operator with the rest of the block — MLP, residuals, normalization — into the full encoder, and examines how the [CLS] register accumulates a sequence summary as depth increases.\n← Previous: Lesson 2 — Tokenization \u0026amp; the Encoder Input · Course index · Next: Lesson 4 — The Encoder Block \u0026amp; Stack → Appendices: A — Encoder Family · B — Encoder vs. Decoder Cheat Sheet · C — Acronyms The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/tutorials/encoder-deep-dive/lesson-03-bidirectional-self-attention/","summary":"\u003ch1 id=\"lesson-3--bidirectional-self-attention\"\u003eLesson 3 — Bidirectional Self-Attention\u003c/h1\u003e\n\u003cp\u003e\u003cem\u003eTechnical course on encoder LLM mechanics — Lesson 3 of 10.\u003c/em\u003e\u003c/p\u003e\n\u003chr\u003e\n\u003ch2 id=\"recap-of-lesson-2\"\u003eRecap of Lesson 2\u003c/h2\u003e\n\u003cp\u003eThe encoder\u0026rsquo;s input tensor $H^{(0)}\\in\\mathbb R^{n\\times d}$ is the sum of three learned embeddings — token (WordPiece, $V=30{,}522$), \u003cstrong\u003esegment\u003c/strong\u003e ($s_i\\in\\{0,1\\}$), and learned-absolute \u003cstrong\u003eposition\u003c/strong\u003e (capped at $n_{\\max}=512$) — with the sequence framed by \u003ccode\u003e[CLS]\u003c/code\u003e, \u003ccode\u003e[SEP]\u003c/code\u003e, \u003ccode\u003e[MASK]\u003c/code\u003e, and \u003ccode\u003e[PAD]\u003c/code\u003e. Each row knows what token it is and where it sits, but nothing about the other tokens. This lesson derives the operator that fixes that, and shows that the \u003cem\u003eonly\u003c/em\u003e thing distinguishing encoder attention from decoder attention is the removal of one mask.\u003c/p\u003e","title":"Bidirectional Self-Attention"},{"content":"Lesson 3 — Self-Attention Technical course on LLM mechanics — Lesson 3 of 10.\nRecap of Lesson 2 The bottom of $f$ maps a string to $H^{(0)}\\in\\mathbb R^{n\\times d}$ via a fixed tokenizer $\\tau$ ([CONFIG], BPE-learned offline) and a learned embedding $E\\in\\mathbb R^{V\\times d}$ ([LEARNED], $Vd$ parameters, row-sparse gradient). Because attention is permutation-equivariant, position is injected separately — sinusoidal, learned, or RoPE (relative position inside the dot product). We now derive the attention operator that turns these independent per-token vectors into contextualized ones, and analyze its cost.\n1. What problem attention solves Start with the picture before the algebra. After Lesson 2, each token has been turned into a vector, but those vectors are computed independently — the vector for \u0026ldquo;it\u0026rdquo; knows nothing about the \u0026ldquo;dog\u0026rdquo; three words earlier that it refers to. The tokens have been embedded but don\u0026rsquo;t yet \u0026ldquo;know\u0026rdquo; about each other. Attention is the mechanism that lets them communicate: every token looks over the whole sequence, decides which other tokens are relevant to it, and pulls in information from them, producing a new, context-aware vector for each position.\nThe machinery below is just a precise way to do that \u0026ldquo;look over, decide relevance, pull in.\u0026rdquo; Each token emits a query (what am I looking for?), every token offers a key (what do I contain?), and the match between a query and a key sets how much value (the actual information) flows from one token to another. Keep that three-role picture in mind and the matrices that follow are bookkeeping for it.\n2. Definition and tensor shapes Let $H\\in\\mathbb R^{n\\times d}$ be the input (rows = tokens). A single attention head is parameterized by three learned projections:\n$$ W_Q,W_K\\in\\mathbb R^{d\\times d_k},\\quad W_V\\in\\mathbb R^{d\\times d_v}, $$producing queries, keys, values\n$$ Q=HW_Q\\in\\mathbb R^{n\\times d_k},\\quad K=HW_K\\in\\mathbb R^{n\\times d_k},\\quad V=HW_V\\in\\mathbb R^{n\\times d_v}. $$Scaled dot-product attention is\n$$ \\boxed{\\ \\mathrm{Attn}(Q,K,V)=\\underbrace{\\mathrm{softmax}\\!\\Big(\\tfrac{QK^\\top}{\\sqrt{d_k}}+M\\Big)}_{A\\in\\mathbb R^{n\\times n}}\\,V\\ \\in\\mathbb R^{n\\times d_v},\\ } $$with softmax applied row-wise and $M$ the mask (below). Shapes: $QK^\\top\\in\\mathbb R^{n\\times n}$, the attention matrix $A\\in\\mathbb R^{n\\times n}$ is row-stochastic ($A\\mathbf 1=\\mathbf 1$, $A_{ij}\\ge0$), and $AV\\in\\mathbb R^{n\\times d_v}$.\nPer-token, output $i$ is a convex combination of value vectors:\n$$ \\mathbf o_i=\\sum_{j=1}^{n}A_{ij}\\,\\mathbf v_j,\\qquad A_{ij}=\\frac{\\exp(s_{ij})}{\\sum_{m}\\exp(s_{im})},\\qquad s_{ij}=\\frac{\\mathbf q_i^\\top\\mathbf k_j}{\\sqrt{d_k}}. $$The score $s_{ij}$ is an inner product in the learned query/key subspace — a learned, asymmetric similarity ($W_Q\\neq W_K$, so \u0026ldquo;$i$ attends to $j$\u0026rdquo; $\\neq$ \u0026ldquo;$j$ attends to $i$\u0026rdquo;).\nThe whole flow for a single head, from input to context-aware output:\nflowchart LR H[\"H (n × d)input token rows\"] H --\u003e|\"× W_Q\"| Q[\"Q (queries)\"] H --\u003e|\"× W_K\"| K[\"K (keys)\"] H --\u003e|\"× W_V\"| V[\"V (values)\"] Q --\u003e S[\"scores S = QKᵀ / √d_k(n × n)\"] K --\u003e S S --\u003e A[\"A = softmax(S + M)row-stochastic weights\"] V --\u003e O[\"output O = A·V(n × d_v)\"] A --\u003e O Read it left to right: the same input $H$ is projected three ways; queries and keys are compared to form a score for every (token $i$, token $j$) pair; softmax turns each row of scores into weights that sum to 1; and each output row is the weighted blend of value vectors those weights pick out.\nA tiny worked example Take three tokens and $d_k=d_v=2$, and follow the query from token 3 as it attends over all three positions. Suppose the projections produced\n$$ \\mathbf q_3=(1,0),\\quad \\mathbf k_1=(1,0),\\ \\mathbf k_2=(0,1),\\ \\mathbf k_3=(1,1),\\quad \\mathbf v_1=(2,0),\\ \\mathbf v_2=(0,2),\\ \\mathbf v_3=(1,1). $$Scores (dot product, then divide by $\\sqrt{d_k}=\\sqrt2\\approx1.414$):\n$$ s_{31}=\\tfrac{\\mathbf q_3\\cdot\\mathbf k_1}{\\sqrt2}=\\tfrac{1}{1.414}=0.707,\\quad s_{32}=\\tfrac{0}{1.414}=0,\\quad s_{33}=\\tfrac{1}{1.414}=0.707. $$Softmax of $(0.707,\\,0,\\,0.707)$: exponentiate to $(2.03,\\,1.00,\\,2.03)$, sum $=5.06$, so the attention weights are\n$$ A_{3,:}=(0.401,\\ 0.198,\\ 0.401),\\qquad \\textstyle\\sum_j A_{3j}=1. $$Token 3 splits its attention mostly between tokens 1 and 3 (the keys its query aligns with) and discounts token 2.\nOutput = weighted blend of the value vectors:\n$$ \\mathbf o_3=0.401\\,(2,0)+0.198\\,(0,2)+0.401\\,(1,1)=(1.20,\\ 0.80). $$Two things to notice. The output is pulled toward $\\mathbf v_1$ and $\\mathbf v_3$ in proportion to the weights — that is \u0026ldquo;pulling in information from relevant tokens.\u0026rdquo; And the $\\sqrt{d_k}$ divisor mattered: without it the scores would be the larger $(1,0,1)$, giving sharper weights $(0.422,0.155,0.422)$; at realistic $d_k=64$–$128$ the unscaled scores would be large enough to saturate softmax toward one-hot and stall the gradient (next section). That divisor looked like a cosmetic constant in the definition; the next section shows it is load-bearing.\nFigure 1. One attention head, end to end.\n3. Why divide by $\\sqrt{d_k}$ — the variance argument This constant is not arbitrary. Model the entries of $\\mathbf q_i,\\mathbf k_j$ as independent, zero-mean, unit-variance random variables. Then the raw score is a sum of $d_k$ independent products:\n$$ \\mathbf q_i^\\top\\mathbf k_j=\\sum_{l=1}^{d_k} q_{il}k_{jl},\\qquad \\mathbb E[\\cdot]=0,\\qquad \\mathrm{Var}[\\cdot]=\\sum_{l=1}^{d_k}\\mathrm{Var}[q_{il}k_{jl}]=d_k. $$So raw logits have standard deviation $\\sqrt{d_k}$, which grows with head dimension. Large-magnitude logits push softmax toward a one-hot output, where its Jacobian (next section) vanishes — gradients die and training stalls. Dividing by $\\sqrt{d_k}$ rescales to unit variance regardless of $d_k$, keeping softmax in a high-gradient regime. It is a fixed [CONFIG] normalizer, not learned.\nThat argument invoked \u0026ldquo;where softmax\u0026rsquo;s gradient vanishes\u0026rdquo; twice without proving it. The Jacobian below makes the claim exact, and we will need its precise form again when the backward pass reaches this layer.\n4. The softmax Jacobian (needed for backprop) For a row with scores $\\mathbf s\\in\\mathbb R^n$ and output $\\mathbf a=\\mathrm{softmax}(\\mathbf s)$, the Jacobian is\n$$ \\frac{\\partial a_i}{\\partial s_j}=a_i(\\delta_{ij}-a_j)\\quad\\Longleftrightarrow\\quad J=\\mathrm{diag}(\\mathbf a)-\\mathbf a\\mathbf a^\\top. $$Two consequences. First, softmax couples all coordinates (the off-diagonal $-a_ia_j$): raising one logit lowers every other probability. Second, when $\\mathbf a$ is near one-hot ($a_i\\to1$ for some $i$, else $\\to0$), $J\\to 0$ — the saturation that the $\\sqrt{d_k}$ scaling exists to avoid. This Jacobian is exactly what the backward pass propagates through (Lesson 7).\n5. Causal masking The operator so far lets every token attend to every other token — but Lesson 1 fixed $f$ as a next-token predictor, and a token that can peek at its own future trivially cheats. Causal masking is the small modification that reconciles the two. For autoregressive modeling, position $i$ must not attend to $j\\gt i$ (that would leak the target). Enforce it with an additive mask before softmax:\n$$ M_{ij}=\\begin{cases}0,\u0026 j\\le i\\\\[2pt] -\\infty,\u0026 j\\gt i\\end{cases} \\quad\\Rightarrow\\quad A_{ij}=0\\ \\text{for } j\\gt i, $$since $\\exp(-\\infty)=0$. The result is a lower-triangular $A$. The mask is a fixed pattern ([CONFIG], zero parameters); it is what makes the model autoregressive and lets all $n$ positions be trained in parallel from a single forward pass while preserving causality.\nFigure 2. The causal mask, before and after softmax.\n6. Multi-head attention We now have one complete, causal attention head. But one head can only express a single notion of relevance at a time, and language needs many at once — syntactic agreement, coreference, position. A single head forces one similarity structure. Multi-head attention runs $h$ heads in parallel, each with its own $\\{W_Q^{(i)},W_K^{(i)},W_V^{(i)}\\}$, then concatenates and projects with $W_O\\in\\mathbb R^{hd_v\\times d}$ ([LEARNED]):\n$$ \\mathrm{MHA}(H)=\\big[\\,\\mathrm{head}_1\\,\\|\\cdots\\|\\,\\mathrm{head}_h\\,\\big]W_O,\\qquad \\mathrm{head}_i=\\mathrm{Attn}(HW_Q^{(i)},HW_K^{(i)},HW_V^{(i)}). $$The standard sizing sets $d_k=d_v=d/h$, so concatenated heads have width $d$ and total attention FLOPs are comparable to a single full-width head. Heads empirically specialize (induction/copying heads, positional heads, syntactic heads), though specialization is emergent, not assigned.\nParameter count (one MHA layer). Four $d\\times d$ matrices (the per-head projections stacked, plus $W_O$): $\\approx 4d^2$ parameters (biases optional). Compare to the MLP\u0026rsquo;s $\\approx 8d^2$ (Lesson 4) — attention is the minority of parameters but the majority of the interesting computation.\n7. Computational and memory complexity We have counted attention\u0026rsquo;s parameters and found them modest. Its runtime cost is a different story, and it is the one that shapes nearly every engineering decision about context length. This is central to everything about long context.\nCompute: $QK^\\top$ is $n\\times d_k$ times $d_k\\times n$ = $O(n^2 d_k)$; $AV$ is likewise $O(n^2 d_v)$. Summed over heads, attention is $\\boxed{O(n^2 d)}$ per layer — quadratic in sequence length. Memory: the materialized $A\\in\\mathbb R^{n\\times n}$ is $O(n^2)$ per head. This is the dominant cost at long $n$ and the reason for IO-aware kernels like FlashAttention, which compute $A$ in tiles without ever writing the full $n\\times n$ matrix to memory (same math, $O(n)$ memory). The quadratic term $n^2 d$ versus the MLP\u0026rsquo;s linear-in-$n$ term $n\\,d^2$ determines the crossover: attention dominates cost once $n\\gtrsim d$.\nLearned / config / state Quantity Category $W_Q,W_K,W_V$ (per head), $W_O$ [LEARNED] $h$, $d_k$, $d_v$ [CONFIG] $\\sqrt{d_k}$ scaling, causal mask $M$ [CONFIG] (no params) Scores $s_{ij}$, attention matrix $A$, outputs $\\mathbf o_i$ [STATE] (recomputed per input) Note $A$ is state, not a parameter: it is a function of the current input, recomputed at inference, never stored in the model.\nFigure 3. Where attention overtakes the MLP.\nConclusion Self-attention computes $\\mathrm{softmax}(QK^\\top/\\sqrt{d_k}+M)V$: each output is a convex combination of value vectors weighted by a learned, asymmetric similarity between query and key projections of the inputs. The $\\sqrt{d_k}$ divisor is a variance normalizer that keeps softmax out of its zero-gradient saturation regime — quantified by the softmax Jacobian $\\mathrm{diag}(\\mathbf a)-\\mathbf a\\mathbf a^\\top$. A $-\\infty$ causal mask enforces autoregression while allowing all positions to train in parallel. Multi-head attention runs $h$ subspace projections in parallel ($d_k=d/h$) and recombines them with $W_O$, costing $\\approx4d^2$ parameters per layer. The defining cost is $O(n^2d)$ compute and $O(n^2)$ memory — quadratic in length — which drives long-context engineering. The learned pieces are the projection matrices; head count, scaling, and masking are config; the attention matrix itself is transient state.\nAttention lets tokens communicate, but on its own it is only one operation; a working model stacks dozens of them, and naively stacking anything dozens deep does not train. Next we wrap attention in the rest of the block — the MLP that holds most parameters, plus the residual and normalization machinery that makes deep stacks trainable — turning a single mixing step into a layer we can repeat $N$ times.\n← Lesson 2 — Tokenization \u0026amp; Embeddings · Course index · Next: Lesson 4 — The Transformer Block → Appendices: A — Acronyms · B — Formulas · C — Attention Variants The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/tutorials/llm-lessons/lesson-03-self-attention/","summary":"\u003ch1 id=\"lesson-3--self-attention\"\u003eLesson 3 — Self-Attention\u003c/h1\u003e\n\u003cp\u003e\u003cem\u003eTechnical course on LLM mechanics — Lesson 3 of 10.\u003c/em\u003e\u003c/p\u003e\n\u003chr\u003e\n\u003ch2 id=\"recap-of-lesson-2\"\u003eRecap of Lesson 2\u003c/h2\u003e\n\u003cp\u003eThe bottom of $f$ maps a string to $H^{(0)}\\in\\mathbb R^{n\\times d}$ via a fixed tokenizer $\\tau$ (\u003cstrong\u003e[CONFIG]\u003c/strong\u003e, BPE-learned offline) and a learned embedding $E\\in\\mathbb R^{V\\times d}$ (\u003cstrong\u003e[LEARNED]\u003c/strong\u003e, $Vd$ parameters, row-sparse gradient). Because attention is permutation-equivariant, position is injected separately — sinusoidal, learned, or RoPE (relative position inside the dot product). We now derive the attention operator that turns these independent per-token vectors into contextualized ones, and analyze its cost.\u003c/p\u003e","title":"Self-Attention"},{"content":"Lesson 3 — The Decoder Stack: Causal Self-Attention Revisited Technical course on encoder–decoder transformers — Lesson 3 of 10.\nRecap of Lesson 2 The encoder reads $x$ with bidirectional self-attention ($M=0$) and emits the memory $H_{\\text{enc}}\\in\\mathbb R^{n\\times d}$ — one contextual vector per input token, computed once, frozen for generation, and tagged [STATE]. It keeps a padding mask so pad positions are never attended to, and passes that mask downstream. Its only deviation from a decoder-only block was a deleted causal mask. We now build the writer, whose self-attention keeps the causal mask but which grows one extra sublayer to consume the memory.\n1. The decoder\u0026rsquo;s job and its two sources of information The decoder generates the output $y=(y_1,\\dots,y_m)$ one token at a time, modeling $P_\\theta(y_t\\mid y_{\\lt t},x)$. To do that it must combine two information streams at every position:\nthe output so far $y_{\\lt t}$ — what it has already written, read causally; the input $x$ — read through the memory $H_{\\text{enc}}$, in full. A decoder-only model has only the first stream (it folds the input into $y_{\\lt t}$ by concatenation). The encoder–decoder decoder keeps the first and adds a separate pathway for the second. That separate pathway is cross-attention, and the need for it is precisely why the decoder block has three sublayers where the encoder block had two. This lesson sets up the causal-self-attention sublayer (which you already know) and the slot the cross-attention sublayer occupies; Lesson 4 derives cross-attention itself.\n2. Causal self-attention, unchanged The decoder\u0026rsquo;s first sublayer is ordinary masked multi-head self-attention over the output sequence — bit-for-bit the mechanism from a decoder-only model. Let $H_{\\text{dec}}\\in\\mathbb R^{m\\times d}$ be the decoder\u0026rsquo;s current hidden states (rows = output positions). Then\n$$ \\mathrm{SelfAttn}(H_{\\text{dec}})=\\mathrm{softmax}\\!\\Big(\\tfrac{Q_sK_s^\\top}{\\sqrt{d_k}}+M^{\\text{causal}}\\Big)V_s, \\quad Q_s,K_s,V_s = H_{\\text{dec}}W_Q^s,\\,H_{\\text{dec}}W_K^s,\\,H_{\\text{dec}}W_V^s, $$with the same $-\\infty$ upper-triangular causal mask as before:\n$$ M^{\\text{causal}}_{ij}=\\begin{cases}0,\u0026 j\\le i\\\\ -\\infty,\u0026 j\\gt i\\end{cases} $$This enforces that output position $i$ attends only to $y_{\\le i}$, which is exactly what makes the output factorization $\\prod_t P_\\theta(y_t\\mid y_{\\lt t},x)$ valid and lets all $m$ output positions be trained in parallel from one forward pass (Lesson 8). Nothing here is new. The superscript $s$ marks these as the self-attention projections, to distinguish them from the cross-attention projections of the next lesson — the decoder owns two complete sets of $W_Q,W_K,W_V$.\n3. Where cross-attention sits: the three-sublayer block A decoder block is the standard two-sublayer block with a cross-attention sublayer inserted in the middle:\n$$ \\begin{aligned} H_1 \u0026= H_{\\text{dec}} + \\mathrm{SelfAttn}_{\\text{causal}}\\big(\\mathrm{Norm}(H_{\\text{dec}})\\big), \u0026\u0026 \\text{(1) read the output so far}\\\\ H_2 \u0026= H_1 + \\mathrm{CrossAttn}\\big(\\mathrm{Norm}(H_1),\\,H_{\\text{enc}}\\big), \u0026\u0026 \\text{(2) read the input memory}\\\\ H_3 \u0026= H_2 + \\mathrm{FFN}\\big(\\mathrm{Norm}(H_2)\\big). \u0026\u0026 \\text{(3) per-position transform} \\end{aligned} $$Each is a pre-norm residual sublayer, so the gradient still flows along an identity path through all three. The ordering is deliberate and standard: the decoder first integrates what it has written (self-attention), then consults the source (cross-attention), then transforms. The cross-attention sublayer is the only one that takes a second argument, $H_{\\text{enc}}$ — the same memory for every block and every step.\nflowchart TB inp[\"H_dec (output positions)\"] --\u003e sa[\"1 · Causal self-attention(over y_","permalink":"https://www.fanfani.net/tutorials/encoder-decoder-lessons/lesson-03-the-decoder-stack/","summary":"\u003ch1 id=\"lesson-3--the-decoder-stack-causal-self-attention-revisited\"\u003eLesson 3 — The Decoder Stack: Causal Self-Attention Revisited\u003c/h1\u003e\n\u003cp\u003e\u003cem\u003eTechnical course on encoder–decoder transformers — Lesson 3 of 10.\u003c/em\u003e\u003c/p\u003e\n\u003chr\u003e\n\u003ch2 id=\"recap-of-lesson-2\"\u003eRecap of Lesson 2\u003c/h2\u003e\n\u003cp\u003eThe encoder reads $x$ with \u003cstrong\u003ebidirectional\u003c/strong\u003e self-attention ($M=0$) and emits the memory $H_{\\text{enc}}\\in\\mathbb R^{n\\times d}$ — one contextual vector per input token, computed once, frozen for generation, and tagged \u003cstrong\u003e[STATE]\u003c/strong\u003e. It keeps a \u003cstrong\u003epadding mask\u003c/strong\u003e so pad positions are never attended to, and passes that mask downstream. Its only deviation from a decoder-only block was a deleted causal mask. We now build the writer, whose self-attention keeps the causal mask but which grows one extra sublayer to consume the memory.\u003c/p\u003e","title":"The Decoder Stack: Causal Self-Attention Revisited"},{"content":"Lesson 3 — The Reverse Process: Learning to Denoise Recap of Lesson 2. The forward process is a fixed Gaussian Markov chain, $q(x_t\\mid x_{t-1}) = \\mathcal{N}(x_t; \\sqrt{1-\\beta_t}\\,x_{t-1}, \\beta_t I)$. With $\\alpha_t = 1-\\beta_t$ and $\\bar\\alpha_t=\\prod_{s\\le t}\\alpha_s$, we proved the closed-form jump $x_t = \\sqrt{\\bar\\alpha_t}\\,x_0 + \\sqrt{1-\\bar\\alpha_t}\\,\\epsilon$, so any noise level is one operation away, and $x_T\\approx\\mathcal N(0,I)$. The noise $\\epsilon$ is the natural training target. Now we build the reverse direction.\nHand a conservator a smudged painting and ask which stroke to undo first, and the honest answer is unsettling: it depends on what the painting was, and that is exactly what you cannot know. But change the rules of the game. Suppose that, while you are still in training, someone quietly lets you see the original. Now the question has a clean, exact answer — you can name the step back to the last coefficient. This lesson is built on that act of permitted cheating: we cannot invert noise out in the open, but conditioned on the clean image we can, and from there we teach a network to manage without the glimpse.\nOverview To generate, we must invert the chain: start at pure noise $x_T\\sim\\mathcal N(0,I)$ and, step by step, sample a slightly-cleaner $x_{t-1}$ from $x_t$. The honest object is the true reverse conditional $q(x_{t-1}\\mid x_t)$. It is intractable: Bayes\u0026rsquo; rule writes it as $q(x_{t-1}\\mid x_t)=q(x_t\\mid x_{t-1})\\,q(x_{t-1})/q(x_t)$, and the marginal $q(x_t)=\\int q(x_t\\mid x_0)\\,q(x_0)\\,dx_0$ depends on the unknown data distribution $q(x_0)$. We therefore (a) approximate the reverse step with a parametric Gaussian $p_\\theta$, justified asymptotically by a result of Feller, and (b) construct a tractable surrogate target by additionally conditioning on $x_0$ — available during training.\nWhy the reverse step is (approximately) Gaussian The functional-form license for modeling each reverse step as a Gaussian is a small-noise (diffusion-limit) statement.\nFeller\u0026rsquo;s result. For a diffusion process with continuous sample paths governed by infinitesimal Gaussian transitions (locally Brownian dynamics with bounded drift and diffusion coefficients), the time-reversed process is itself a diffusion with Gaussian infinitesimal transitions (Feller 1949; made precise for the reverse-time SDE by Anderson 1982, which we revisit in Lesson 9). Concretely, as the step size $\\beta_t\\to 0$, the transition $q(x_{t-1}\\mid x_t)$ converges to a Gaussian whose mean is $x_t$ plus an $O(\\beta_t)$ score-driven correction and whose covariance is $\\beta_t I + o(\\beta_t)$.\nRigor remark. This is an asymptotic, small-step statement, not an exact identity for finite $\\beta_t$. For finite step size the true $q(x_{t-1}\\mid x_t)$ is a Gaussian mixture (an $x_0$-average of the tractable posteriors derived below), hence generally non-Gaussian and multimodal. The Gaussian parametrization of $p_\\theta$ incurs an $O(\\beta_t^2)$ per-step bias; the bias accumulates over $T$ steps but vanishes in the joint limit $T\\to\\infty$, $\\max_t\\beta_t\\to0$ with $\\bar\\alpha_T$ held fixed. This is the formal reason the schedule uses many small steps: it is what makes the Gaussian reverse model consistent, not merely a quality heuristic.\nWe therefore posit\n$$ p_\\theta(x_{t-1}\\mid x_t) = \\mathcal{N}\\!\\left(x_{t-1};\\ \\mu_\\theta(x_t, t),\\ \\Sigma_\\theta(x_t, t)\\right), $$and define the full reverse (generative) model as the Markov chain\n$$ p_\\theta(x_{0:T}) = p(x_T)\\prod_{t=1}^{T} p_\\theta(x_{t-1}\\mid x_t), \\qquad p(x_T) = \\mathcal{N}(0, I). $$A single network, shared across $t$ and conditioned on $t$ (via a time embedding), emits the mean $\\mu_\\theta(x_t,t)$ and optionally the covariance.\nThe tractable posterior The honest object is out of reach, so we strike a bargain: we will solve a nearby problem we can write down, and arrange for the network to inherit its answer.\nWhile $q(x_{t-1}\\mid x_t)$ is intractable, the forward posterior $q(x_{t-1}\\mid x_t,x_0)$ — conditioning additionally on the clean image — is exactly Gaussian and available in closed form. During training we hold $x_0$, so we use $q(x_{t-1}\\mid x_t,x_0)$ as the ground-truth regression target for $p_\\theta(x_{t-1}\\mid x_t)$.\nBy Bayes\u0026rsquo; rule, and the forward Markov property $q(x_t\\mid x_{t-1},x_0)=q(x_t\\mid x_{t-1})$:\n$$ q(x_{t-1}\\mid x_t, x_0) = q(x_t\\mid x_{t-1})\\,\\frac{q(x_{t-1}\\mid x_0)}{q(x_t\\mid x_0)}. $$Every factor is a known Gaussian (Lesson 2). A product/ratio of Gaussian densities in $x_{t-1}$ is Gaussian, so\n$$ q(x_{t-1}\\mid x_t, x_0) = \\mathcal{N}\\!\\left(x_{t-1};\\ \\tilde\\mu_t(x_t, x_0),\\ \\tilde\\beta_t I\\right). $$Full derivation of $\\tilde\\beta_t$ and $\\tilde\\mu_t$ We complete the square. The three relevant densities are\n$$ q(x_t\\mid x_{t-1})=\\mathcal N\\!\\big(\\sqrt{\\alpha_t}\\,x_{t-1},\\,\\beta_t I\\big),\\quad q(x_{t-1}\\mid x_0)=\\mathcal N\\!\\big(\\sqrt{\\bar\\alpha_{t-1}}\\,x_0,\\,(1-\\bar\\alpha_{t-1})I\\big), $$and $q(x_t\\mid x_0)$, which does not depend on $x_{t-1}$ and is absorbed into the normalizer. Working component-wise (the covariances are isotropic, so each coordinate decouples), the log of the unnormalized posterior, keeping only $x_{t-1}$-dependent terms, is\n$$ \\log q(x_{t-1}\\mid x_t,x_0) = -\\frac{1}{2}\\left[\\frac{(x_t-\\sqrt{\\alpha_t}\\,x_{t-1})^2}{\\beta_t} +\\frac{(x_{t-1}-\\sqrt{\\bar\\alpha_{t-1}}\\,x_0)^2}{1-\\bar\\alpha_{t-1}}\\right]+\\text{const}. $$Expand both squares and group powers of $x_{t-1}$:\n$$ = -\\frac{1}{2}\\left[\\left(\\frac{\\alpha_t}{\\beta_t}+\\frac{1}{1-\\bar\\alpha_{t-1}}\\right)x_{t-1}^2\n2\\left(\\frac{\\sqrt{\\alpha_t}}{\\beta_t},x_t + \\frac{\\sqrt{\\bar\\alpha_{t-1}}}{1-\\bar\\alpha_{t-1}},x_0\\right)x_{t-1}\\right]+\\text{const}. $$ Matching to the canonical Gaussian exponent $-\\tfrac{1}{2}\\big(x_{t-1}^2/\\tilde\\beta_t - 2\\,\\tilde\\mu_t\\, x_{t-1}/\\tilde\\beta_t\\big)$ identifies the precision (coefficient of $x_{t-1}^2$):\n$$ \\frac{1}{\\tilde\\beta_t} = \\frac{\\alpha_t}{\\beta_t} + \\frac{1}{1-\\bar\\alpha_{t-1}} = \\frac{\\alpha_t(1-\\bar\\alpha_{t-1})+\\beta_t}{\\beta_t\\,(1-\\bar\\alpha_{t-1})}. $$Verifying $1-\\bar\\alpha_t = \\alpha_t(1-\\bar\\alpha_{t-1})+\\beta_t$. Expand the right side using $\\beta_t=1-\\alpha_t$ and $\\bar\\alpha_t=\\alpha_t\\bar\\alpha_{t-1}$:\n$$ \\alpha_t(1-\\bar\\alpha_{t-1})+\\beta_t = \\alpha_t - \\alpha_t\\bar\\alpha_{t-1} + (1-\\alpha_t) = 1 - \\alpha_t\\bar\\alpha_{t-1} = 1-\\bar\\alpha_t. \\quad\\checkmark $$Hence the numerator is $1-\\bar\\alpha_t$, and inverting the precision gives\n$$ \\boxed{\\ \\tilde\\beta_t = \\frac{1-\\bar\\alpha_{t-1}}{1-\\bar\\alpha_t}\\,\\beta_t\\ } $$The mean equals (linear coefficient) $\\times\\,\\tilde\\beta_t$:\n$$ \\tilde\\mu_t = \\tilde\\beta_t\\left(\\frac{\\sqrt{\\alpha_t}}{\\beta_t}\\,x_t + \\frac{\\sqrt{\\bar\\alpha_{t-1}}}{1-\\bar\\alpha_{t-1}}\\,x_0\\right). $$Substitute $\\tilde\\beta_t=\\dfrac{(1-\\bar\\alpha_{t-1})\\beta_t}{1-\\bar\\alpha_t}$ and simplify each term. For the $x_t$ term, $\\dfrac{(1-\\bar\\alpha_{t-1})\\beta_t}{1-\\bar\\alpha_t}\\cdot\\dfrac{\\sqrt{\\alpha_t}}{\\beta_t}=\\dfrac{\\sqrt{\\alpha_t}(1-\\bar\\alpha_{t-1})}{1-\\bar\\alpha_t}$. For the $x_0$ term, $\\dfrac{(1-\\bar\\alpha_{t-1})\\beta_t}{1-\\bar\\alpha_t}\\cdot\\dfrac{\\sqrt{\\bar\\alpha_{t-1}}}{1-\\bar\\alpha_{t-1}}=\\dfrac{\\sqrt{\\bar\\alpha_{t-1}}\\,\\beta_t}{1-\\bar\\alpha_t}$. Therefore\n$$ \\boxed{\\ \\tilde\\mu_t(x_t,x_0) = \\frac{\\sqrt{\\bar\\alpha_{t-1}}\\,\\beta_t}{1-\\bar\\alpha_t}\\,x_0 + \\frac{\\sqrt{\\alpha_t}\\,(1-\\bar\\alpha_{t-1})}{1-\\bar\\alpha_t}\\,x_t\\ } $$The two coefficients sum to one only in the noiseless limit; in general the posterior mean is an affine blend of $x_0$ and $x_t$ with schedule-determined weights, plus the variance $\\tilde\\beta_t\\lt \\beta_t$ (knowing $x_0$ sharpens the step).\nRigor remark. $\\tilde\\beta_t$ and $\\tilde\\mu_t$ depend only on the schedule and on $(x_t,x_0)$, not on $\\theta$. They are exact for every finite $\\beta_t$; the Gaussian family of the posterior is exact here precisely because we conditioned on $x_0$ (the marginalized version is the mixture noted earlier). At $t=1$, $\\bar\\alpha_0:=1$, so $\\tilde\\beta_1=0$ and the posterior is a point mass — handled by the discrete decoder $L_0$ in Lesson 4.\nFigure 1. What conditioning on x₀ is worth: the posterior variance.\nRewriting the target in terms of noise The closed-form jump is invertible:\n$$ x_t = \\sqrt{\\bar\\alpha_t}\\,x_0 + \\sqrt{1-\\bar\\alpha_t}\\,\\epsilon \\quad\\Longrightarrow\\quad x_0 = \\frac{1}{\\sqrt{\\bar\\alpha_t}}\\left(x_t - \\sqrt{1-\\bar\\alpha_t}\\,\\epsilon\\right). $$Substitute this into $\\tilde\\mu_t$ and carry the algebra through. Insert $x_0$ into the boxed mean:\n$$ \\tilde\\mu_t = \\frac{\\sqrt{\\bar\\alpha_{t-1}},\\beta_t}{1-\\bar\\alpha_t}\\cdot\\frac{1}{\\sqrt{\\bar\\alpha_t}}\\big(x_t-\\sqrt{1-\\bar\\alpha_t},\\epsilon\\big)\n\\frac{\\sqrt{\\alpha_t},(1-\\bar\\alpha_{t-1})}{1-\\bar\\alpha_t},x_t. $$ Use $\\bar\\alpha_t=\\alpha_t\\bar\\alpha_{t-1}$, so $\\dfrac{\\sqrt{\\bar\\alpha_{t-1}}}{\\sqrt{\\bar\\alpha_t}}=\\dfrac{1}{\\sqrt{\\alpha_t}}$. The first prefactor becomes $\\dfrac{\\beta_t}{(1-\\bar\\alpha_t)\\sqrt{\\alpha_t}}$. Collect the coefficient of $x_t$:\n$$ \\frac{\\beta_t}{(1-\\bar\\alpha_t)\\sqrt{\\alpha_t}} + \\frac{\\sqrt{\\alpha_t}(1-\\bar\\alpha_{t-1})}{1-\\bar\\alpha_t} = \\frac{1}{\\sqrt{\\alpha_t}\\,(1-\\bar\\alpha_t)}\\Big[\\beta_t + \\alpha_t(1-\\bar\\alpha_{t-1})\\Big] = \\frac{1-\\bar\\alpha_t}{\\sqrt{\\alpha_t}\\,(1-\\bar\\alpha_t)} = \\frac{1}{\\sqrt{\\alpha_t}}, $$where the bracket collapsed via the identity $\\beta_t+\\alpha_t(1-\\bar\\alpha_{t-1})=1-\\bar\\alpha_t$ proved above. The coefficient of $\\epsilon$ is\n$$ -\\frac{\\beta_t}{(1-\\bar\\alpha_t)\\sqrt{\\alpha_t}}\\cdot\\sqrt{1-\\bar\\alpha_t} = -\\frac{\\beta_t}{\\sqrt{\\alpha_t}\\,\\sqrt{1-\\bar\\alpha_t}}. $$Therefore\n$$ \\boxed{\\ \\tilde\\mu_t(x_t, x_0) = \\frac{1}{\\sqrt{\\alpha_t}}\\left(x_t - \\frac{\\beta_t}{\\sqrt{1-\\bar\\alpha_t}}\\,\\epsilon\\right)\\ } $$The optimal reverse mean depends on $x_t$ (available at inference) and on the noise $\\epsilon$ that produced it (unknown at inference). If a network predicts that noise, $\\epsilon_\\theta(x_t,t)\\approx\\epsilon$, it recovers the ideal denoising mean by substituting $\\epsilon_\\theta$ in the box. This is why DDPM trains a noise predictor (Lesson 5).\nFigure 2. The same mean, written twice — and why the second form matters.\nFour equivalent parameterizations The same reverse step can be carried by any of four different couriers; which one you hire turns out to matter far more for the training than for the truth.\nThe reverse mean can be carried by any of four equivalent targets, related by invertible linear maps (at fixed $x_t,t$). Let $a_t=\\sqrt{\\bar\\alpha_t}$, $b_t=\\sqrt{1-\\bar\\alpha_t}$, so $x_t=a_t x_0 + b_t\\epsilon$.\n$\\mu$-prediction. The network outputs $\\mu_\\theta(x_t,t)$ directly. Most general but empirically harder to optimize; couples poorly across noise levels. $x_0$-prediction. Output $\\hat x_0(x_t,t)$, then set $\\mu_\\theta=\\tilde\\mu_t(x_t,\\hat x_0)$. Map to noise: $\\hat\\epsilon=(x_t-a_t\\hat x_0)/b_t$. Intuitive, but high-variance loss at large $t$ (small $a_t$). $\\epsilon$-prediction (DDPM standard). Output $\\epsilon_\\theta(x_t,t)$, set $\\mu_\\theta=\\frac{1}{\\sqrt{\\alpha_t}}\\big(x_t-\\frac{\\beta_t}{\\sqrt{1-\\bar\\alpha_t}}\\epsilon_\\theta\\big)$. Map to $x_0$: $\\hat x_0=(x_t-b_t\\epsilon_\\theta)/a_t$. Cleanest loss (Lesson 5). $v$-prediction (Salimans \u0026amp; Ho 2022). Target $v\\equiv a_t\\epsilon - b_t x_0$. Inverse maps: $x_0=a_t x_t - b_t v$, $\\epsilon=b_t x_t + a_t v$. The map $(x_0,\\epsilon)\\mapsto(x_t,v)$ is the rotation $\\left(\\begin{smallmatrix}a_t \u0026 b_t\\\\ -b_t \u0026 a_t\\end{smallmatrix}\\right)$ (since $a_t^2+b_t^2=1$). Behaves well across all SNR, especially near $t=T$ and in distillation. Rigor remark. All four are exactly equivalent for the optimal predictor and differ only in the loss geometry / implicit per-$t$ weighting they induce; they are not equivalent under a fixed network and finite data. Choosing a parameterization is choosing a reweighting of the per-step objective (Lesson 5).\n# Schedule precompute (float64 for stability) and posterior coefficients. import numpy as np betas = np.linspace(1e-4, 2e-2, 1000, dtype=np.float64) # (T,) alphas = 1.0 - betas abar = np.cumprod(alphas) # bar_alpha_t, (T,) abar_prev = np.concatenate([[1.0], abar[:-1]]) # bar_alpha_{t-1} post_var = betas * (1.0 - abar_prev) / (1.0 - abar) # tilde_beta_t # tilde_mu coefficients (eq. for tilde_mu_t(x_t, x_0)): coef_x0 = np.sqrt(abar_prev) * betas / (1.0 - abar) coef_xt = np.sqrt(alphas) * (1.0 - abar_prev) / (1.0 - abar) # Posterior log-variance is clipped at t=1 since tilde_beta_1 = 0: post_logvar = np.log(np.maximum(post_var, 1e-20)) Numerical note. Compute $\\bar\\alpha_t$ in float64 and clip $\\tilde\\beta_t$ before taking logs ($\\tilde\\beta_1=0$). Prefer $\\log(1-\\bar\\alpha_t)$ via log1p(-abar) when $\\bar\\alpha_t$ is close to 1 (small $t$) to avoid catastrophic cancellation.\nFigure 3. Four couriers for one reverse step.\nExercises 1. Complete the square yourself. From the two relevant log-densities, read off the coefficient of $x_{t-1}^2$ and show the posterior precision is $1/\\tilde\\beta_t=\\alpha_t/\\beta_t+1/(1-\\bar\\alpha_{t-1})$, then invert to recover $\\tilde\\beta_t=\\frac{1-\\bar\\alpha_{t-1}}{1-\\bar\\alpha_t}\\beta_t$.\nThe term $-\\frac{(x_t-\\sqrt{\\alpha_t}x_{t-1})^2}{2\\beta_t}$ contributes $\\frac{\\alpha_t}{\\beta_t}$ to the coefficient of $x_{t-1}^2$; the term $-\\frac{(x_{t-1}-\\sqrt{\\bar\\alpha_{t-1}}x_0)^2}{2(1-\\bar\\alpha_{t-1})}$ contributes $\\frac{1}{1-\\bar\\alpha_{t-1}}$. Their sum is the precision $1/\\tilde\\beta_t$. Over the common denominator $\\beta_t(1-\\bar\\alpha_{t-1})$ the numerator is $\\alpha_t(1-\\bar\\alpha_{t-1})+\\beta_t=1-\\bar\\alpha_t$ (the identity proved in the text), so $1/\\tilde\\beta_t=\\frac{1-\\bar\\alpha_t}{\\beta_t(1-\\bar\\alpha_{t-1})}$, giving $\\tilde\\beta_t=\\frac{1-\\bar\\alpha_{t-1}}{1-\\bar\\alpha_t}\\beta_t$.\n2. Posterior variance ordering. Show $\\tilde\\beta_t\\le\\beta_t$ for all $t$, and identify where the ratio approaches $1$ and where the gap is largest.\n$\\dfrac{\\tilde\\beta_t}{\\beta_t}=\\dfrac{1-\\bar\\alpha_{t-1}}{1-\\bar\\alpha_t}$. Since $\\bar\\alpha_t=\\alpha_t\\bar\\alpha_{t-1}\\le\\bar\\alpha_{t-1}$ (because $\\alpha_t\\le1$), we have $1-\\bar\\alpha_t\\ge1-\\bar\\alpha_{t-1}$, so the ratio is $\\le1$: $\\tilde\\beta_t\\le\\beta_t$. As $t\\to T$ both $\\bar\\alpha$\u0026rsquo;s $\\to0$ and the ratio $\\to1$. The gap is largest at small $t$; at $t=1$, $\\bar\\alpha_0:=1$ gives $\\tilde\\beta_1=0$ (knowing $x_0$ pins $x_0$ exactly — a point mass).\n3. From $x_0$-form to noise-form. Substitute $x_0=\\frac{1}{\\sqrt{\\bar\\alpha_t}}(x_t-\\sqrt{1-\\bar\\alpha_t}\\,\\epsilon)$ into $\\tilde\\mu_t(x_t,x_0)=\\frac{\\sqrt{\\bar\\alpha_{t-1}}\\beta_t}{1-\\bar\\alpha_t}x_0+\\frac{\\sqrt{\\alpha_t}(1-\\bar\\alpha_{t-1})}{1-\\bar\\alpha_t}x_t$ and verify the coefficient of $x_t$ collapses to $1/\\sqrt{\\alpha_t}$.\nUsing $\\sqrt{\\bar\\alpha_{t-1}}/\\sqrt{\\bar\\alpha_t}=1/\\sqrt{\\alpha_t}$, the $x_0$-term contributes $\\frac{\\beta_t}{(1-\\bar\\alpha_t)\\sqrt{\\alpha_t}}x_t$ (plus an $\\epsilon$ piece). Adding the existing $x_t$-coefficient:\n$$ \\frac{\\beta_t}{(1-\\bar\\alpha_t)\\sqrt{\\alpha_t}}+\\frac{\\sqrt{\\alpha_t}(1-\\bar\\alpha_{t-1})}{1-\\bar\\alpha_t} =\\frac{\\beta_t+\\alpha_t(1-\\bar\\alpha_{t-1})}{(1-\\bar\\alpha_t)\\sqrt{\\alpha_t}} =\\frac{1-\\bar\\alpha_t}{(1-\\bar\\alpha_t)\\sqrt{\\alpha_t}}=\\frac{1}{\\sqrt{\\alpha_t}}, $$using $\\beta_t+\\alpha_t(1-\\bar\\alpha_{t-1})=1-\\bar\\alpha_t$. The $\\epsilon$-coefficient works out to $-\\frac{\\beta_t}{\\sqrt{\\alpha_t}\\sqrt{1-\\bar\\alpha_t}}$, giving the boxed noise-form mean.\n4. Converting between targets. Given a network output $\\hat\\epsilon$ at $(x_t,t)$, and shorthand $a_t=\\sqrt{\\bar\\alpha_t},\\,b_t=\\sqrt{1-\\bar\\alpha_t}$, express $\\hat x_0$ and the velocity $v$ purely in terms of $\\hat\\epsilon$ and $x_t$.\nFrom $x_t=a_t x_0+b_t\\epsilon$: $\\hat x_0=(x_t-b_t\\hat\\epsilon)/a_t$. For $v=a_t\\epsilon-b_t x_0$, substitute $x_0$ and use $a_t^2+b_t^2=1$:\n$$ v=a_t\\hat\\epsilon-b_t\\frac{x_t-b_t\\hat\\epsilon}{a_t}=\\frac{(a_t^2+b_t^2)\\hat\\epsilon-b_t x_t}{a_t}=\\frac{\\hat\\epsilon-b_t x_t}{a_t}. $$(Equivalently $\\epsilon=b_t x_t+a_t v$, the inverse map of Lesson 5.)\nConclusion The reverse step is Gaussian asymptotically, because the forward steps are infinitesimal (Feller); for finite $\\beta_t$ this is an $O(\\beta_t^2)$ approximation, which is why $T$ is large. We cannot access the true reverse conditional, but conditioning on $x_0$ yields the exact tractable posterior $q(x_{t-1}\\mid x_t,x_0)=\\mathcal N(\\tilde\\mu_t,\\tilde\\beta_t I)$, derived in full by completing the square. Substituting $x_0=x_0(x_t,\\epsilon)$ gave the noise-form mean, showing that predicting $\\epsilon$ suffices to recover the ideal denoiser. We catalogued the four parameterizations and the linear maps between them. Lesson 4 derives why matching this posterior is the principled objective via the variational lower bound.\nThe glimpse of $x_0$ bought us an exact reverse step to imitate, and a clean reason to predict noise. But \u0026ldquo;imitate this posterior\u0026rdquo; is still, at this point, an assertion. Lesson 4 supplies the justification — it shows that matching this step is not a convenient hack but the unavoidable consequence of pushing up a bound on the data\u0026rsquo;s likelihood.\nReferences J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, S. Ganguli. Deep Unsupervised Learning using Nonequilibrium Thermodynamics. ICML 2015. J. Ho, A. Jain, P. Abbeel. Denoising Diffusion Probabilistic Models. NeurIPS 2020. W. Feller. On the Theory of Stochastic Processes, with Particular Reference to Applications. Proc. Berkeley Symp. on Math. Stat. and Prob., 1949 (Gaussianity of reverse infinitesimal transitions). B. D. O. Anderson. Reverse-Time Diffusion Equation Models. Stochastic Processes and their Applications, 1982 (reverse-time SDE; forward pointer to Lesson 9). T. Salimans, J. Ho. Progressive Distillation for Fast Sampling of Diffusion Models. ICLR 2022 ($v$-parameterization). ← Previous: Lesson 2 — The Forward Process · Course Index · Next: Lesson 4 — The Variational Lower Bound → · Appendix A: Acronyms · Appendix B: Formulas The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/tutorials/diffusion-lessons/lesson-03-reverse-process/","summary":"\u003ch1 id=\"lesson-3--the-reverse-process-learning-to-denoise\"\u003eLesson 3 — The Reverse Process: Learning to Denoise\u003c/h1\u003e\n\u003cblockquote\u003e\n\u003cp\u003e\u003cstrong\u003eRecap of Lesson 2.\u003c/strong\u003e The forward process is a fixed Gaussian Markov chain, $q(x_t\\mid x_{t-1}) = \\mathcal{N}(x_t; \\sqrt{1-\\beta_t}\\,x_{t-1}, \\beta_t I)$. With $\\alpha_t = 1-\\beta_t$ and $\\bar\\alpha_t=\\prod_{s\\le t}\\alpha_s$, we proved the closed-form jump $x_t = \\sqrt{\\bar\\alpha_t}\\,x_0 + \\sqrt{1-\\bar\\alpha_t}\\,\\epsilon$, so any noise level is one operation away, and $x_T\\approx\\mathcal N(0,I)$. The noise $\\epsilon$ is the natural training target. Now we build the \u003cem\u003ereverse\u003c/em\u003e direction.\u003c/p\u003e","title":"The Reverse Process"},{"content":"Lesson 3 — The Router / Gating Network Technical course on MoE LLM mechanics — Lesson 3 of 10.\nRecap of Lesson 2 An MoE layer replicates the dense FFN into $E$ experts $E_i(\\mathbf x)$ and combines a selected $k$ of them as $\\mathrm{MoE}(\\mathbf x)=\\sum_{i\\in\\mathcal T_k(\\mathbf x)}g_i(\\mathbf x)E_i(\\mathbf x)$, dropped into the same residual block. The experts hold essentially all the parameters, so $P_{\\text{tot}}$ scales with $E$ while $P_{\\text{act}}$ stays pinned to $k$. Two pieces were assumed but not built: the gate weights $g_i(\\mathbf x)$ and the selected set $\\mathcal T_k(\\mathbf x)$. Both come from one small network — the router — which this lesson constructs.\nThe router is the only genuinely new learned mechanism in the architecture. It is also the smallest component and, paradoxically, the one that causes nearly all of MoE\u0026rsquo;s training difficulty. This lesson builds the router\u0026rsquo;s forward pass; Lessons 4 and 5 deal with its gradient and its pathologies.\n1. The router is a linear classifier over experts For each token representation $\\mathbf x\\in\\mathbb R^d$ (the normalized residual entering the MoE sub-layer), the router computes a score for every expert with a single learned matrix $W_g\\in\\mathbb R^{E\\times d}$ ([LEARNED]):\n$$ \\mathbf h(\\mathbf x)=W_g\\,\\mathbf x\\ \\in\\mathbb R^{E}, $$the vector of router logits — one number per expert, \u0026ldquo;how much does this token want expert $i$.\u0026rdquo; That is the whole router: an $E\\times d$ matrix, $Ed$ parameters, no bias and no nonlinearity in the basic form. Against the experts\u0026rsquo; $\\approx 8Ed^2$ parameters, the router is a rounding error — yet it determines which of those parameters ever run.\nThe logits become probabilities through a softmax, exactly as the model\u0026rsquo;s output head does over the vocabulary:\n$$ P_i(\\mathbf x)=\\frac{e^{h_i(\\mathbf x)}}{\\sum_{j=1}^{E} e^{h_j(\\mathbf x)}},\\qquad \\mathbf P(\\mathbf x)\\in\\Delta^{E-1}. $$$\\mathbf P(\\mathbf x)$ is a categorical distribution over experts — the router\u0026rsquo;s belief about which expert should handle this token. We now turn that soft distribution into a hard, sparse selection.\nFigure 1. The router: one matrix, a softmax, and a hard cut.\n2. Top-k selection: from soft distribution to sparse gate Let $\\mathcal T_k(\\mathbf x)=\\operatorname{arg\\,top\\text-}k_i\\, h_i(\\mathbf x)$ be the indices of the $k$ largest logits. Only these experts run. The gate weights $g_i$ are the routing probabilities restricted to that set and renormalized so they sum to one:\n$$ g_i(\\mathbf x)= \\begin{cases} \\dfrac{e^{h_i(\\mathbf x)}}{\\sum_{j\\in\\mathcal T_k(\\mathbf x)} e^{h_j(\\mathbf x)}}, \u0026 i\\in\\mathcal T_k(\\mathbf x),\\\\[2mm] 0, \u0026 \\text{otherwise.} \\end{cases} $$These $g_i$ are exactly the coefficients in the Lesson 2 combine. With $k=1$ (Switch Transformer) the single chosen expert gets weight $g=1$ — although in practice the value of that one routing probability is still kept as a multiplier (Lesson 4) so a gradient can reach the router. With $k=2$ (Mixtral, GShard) the two chosen experts get convex weights summing to one.\nsoftmax-before-top-k vs. softmax-after-top-k A subtle but real [CONFIG] choice is when the softmax is applied:\nSoftmax then top-$k$ (renormalize): compute $\\mathbf P$ over all $E$ experts, take the top-$k$ entries, renormalize them. The gate weights reflect each chosen expert\u0026rsquo;s share of the full distribution\u0026rsquo;s mass. Top-$k$ then softmax: take the top-$k$ logits first, softmax only over those $k$. The weights depend only on the relative logits among the winners. The two give different gradients to the router and slightly different dynamics; modern models pick one and stay consistent. The renormalize-the-top-$k$ form (shown above) is the common default.\nFigure 2. What top-k assignment looks like over a batch.\n3. Noisy top-k gating: encouraging exploration The original sparse-MoE paper (Shazeer et al., 2017) added tunable Gaussian noise to the logits before the top-$k$, to keep routing from prematurely committing to a fixed set of experts and to spread load during early training:\n$$ h_i(\\mathbf x)=\\big(W_g\\mathbf x\\big)_i+\\varepsilon_i\\cdot \\mathrm{softplus}\\big((W_{\\text{noise}}\\mathbf x)_i\\big),\\qquad \\varepsilon_i\\sim\\mathcal N(0,1). $$The noise scale is itself token-dependent and learned through a second small matrix $W_{\\text{noise}}$ ([LEARNED]). Noise makes the which-experts decision stochastic in early training — a form of exploration that helps every expert receive some tokens before the router hardens its preferences. The noise $\\varepsilon$ is sampled fresh per token per step ([STATE]) and is typically dropped (set to zero) at inference, making routing deterministic at serving time. Many later models (Switch, Mixtral) drop the noise entirely and rely on the auxiliary load-balancing loss of Lesson 5 instead; noisy gating is one of two historical tools for the same problem.\nFigure 3. Two historical tools for one problem, and what neither fixes.\n4. The discrete-choice problem, previewed The top-$k$ operation is the crux of MoE\u0026rsquo;s difficulty. $\\arg\\text{top-}k$ is a discrete, piecewise-constant function of the logits: nudging a logit usually does not change which experts are selected, so $\\partial\\mathcal T_k/\\partial \\mathbf h=0$ almost everywhere. A discrete decision has no useful gradient — yet the router must be trained by gradient descent like everything else.\nThe resolution, derived fully in Lesson 4, is that the router is trained through the gate weights $g_i$, not through the selection $\\mathcal T_k$. The selected experts\u0026rsquo; outputs are multiplied by the continuous probabilities $g_i(\\mathbf x)$, so the loss depends differentiably on the logits of the chosen experts even though the choice of which experts is discrete. The router learns \u0026ldquo;of the experts I picked, weight the good ones more\u0026rdquo;; it does not get a direct gradient saying \u0026ldquo;you should have picked a different expert.\u0026rdquo; That asymmetry — differentiable weighting, non-differentiable selection — is why a separate, explicit balancing loss (Lesson 5) is needed to shape which experts get chosen.\nContrast with the dense model. A dense FFN has no decision to make: every token takes the same path, and every parameter receives a gradient on every token. The MoE router introduces a choice, and a choice is discrete. Nothing in a dense transformer is discrete in this way — the entire dense forward pass is smooth in $\\theta$. The router is the one place an LLM\u0026rsquo;s forward pass makes a hard, combinatorial decision, and the machinery of Lessons 4–6 exists solely to train and to balance that decision.\n5. Learned / config / state Quantity Category Router matrix $W_g$ (and $W_{\\text{noise}}$ if used) [LEARNED] $k$, softmax-before/after, use of noise, router precision (Lesson 8) [CONFIG] Logits $\\mathbf h$, distribution $\\mathbf P$, gate weights $\\mathbf g$, selection $\\mathcal T_k$, noise samples $\\varepsilon$ [STATE] Conclusion The router is a single learned matrix $W_g\\in\\mathbb R^{E\\times d}$ that scores every expert for each token, $\\mathbf h=W_g\\mathbf x$; a softmax turns the scores into a distribution over experts $\\mathbf P=\\mathrm{softmax}(\\mathbf h)$, and top-$k$ selection keeps the $k$ largest, renormalizing them into the gate weights $g_i$ that combine the chosen experts. Noisy top-$k$ gating optionally perturbs the logits to encourage early exploration. The router is tiny ($Ed$ parameters) but pivotal, and it introduces the one thing a dense transformer never has — a discrete decision ($\\arg\\text{top-}k$, with zero gradient almost everywhere). Because the selection is non-differentiable, the router is trained only through the continuous gate weights of the experts it did pick; it gets no direct signal to pick differently. That gap is exactly what makes load balancing a separate, explicit problem.\nNext we put the router and experts together into the full sparse forward pass and trace the gradient back through it — seeing precisely how, and how little, the router learns.\n← Lesson 2 — From the Dense FFN to a Layer of Experts · Course index · Next: Lesson 4 — Sparse Top-k Computation \u0026amp; the Combine → Appendices: A — MoE Family · B — Dense vs. MoE · C — Acronyms The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/tutorials/moe-lessons/lesson-03-the-router-and-gating/","summary":"\u003ch1 id=\"lesson-3--the-router--gating-network\"\u003eLesson 3 — The Router / Gating Network\u003c/h1\u003e\n\u003cp\u003e\u003cem\u003eTechnical course on MoE LLM mechanics — Lesson 3 of 10.\u003c/em\u003e\u003c/p\u003e\n\u003chr\u003e\n\u003ch2 id=\"recap-of-lesson-2\"\u003eRecap of Lesson 2\u003c/h2\u003e\n\u003cp\u003eAn MoE layer replicates the dense FFN into $E$ experts $E_i(\\mathbf x)$ and combines a selected $k$ of them as $\\mathrm{MoE}(\\mathbf x)=\\sum_{i\\in\\mathcal T_k(\\mathbf x)}g_i(\\mathbf x)E_i(\\mathbf x)$, dropped into the same residual block. The experts hold essentially all the parameters, so $P_{\\text{tot}}$ scales with $E$ while $P_{\\text{act}}$ stays pinned to $k$. Two pieces were assumed but not built: the gate weights $g_i(\\mathbf x)$ and the selected set $\\mathcal T_k(\\mathbf x)$. Both come from one small network — the \u003cstrong\u003erouter\u003c/strong\u003e — which this lesson constructs.\u003c/p\u003e","title":"The Router and Gating Network"},{"content":"Lesson 4 — Cross-Attention: The Only New Mechanism Technical course on encoder–decoder transformers — Lesson 4 of 10.\nRecap of Lesson 3 The decoder block has three pre-norm residual sublayers: causal self-attention over the output so far, then cross-attention to the encoder memory, then an FFN. The first and last are inherited unchanged from a decoder-only model; the middle one is the single new operation in the whole architecture, and the decoder carries a second full set of attention projections to run it. We deferred its derivation twice. This lesson pays the debt: cross-attention is scaled dot-product attention with one change — its queries come from the decoder, its keys and values from the encoder.\n1. The idea: attend to a different sequence Self-attention computes $Q,K,V$ all from the same input $H$. The entire derivation never required that. Scaled dot-product attention is defined for any three matrices $Q,K,V$ with compatible shapes; the \u0026ldquo;self\u0026rdquo; in self-attention is the special case $Q,K,V=HW_Q,HW_K,HW_V$. Cross-attention is the general case: let the queries come from one sequence and the keys/values from another.\nIn the decoder, the query sequence is the decoder\u0026rsquo;s own hidden states $H_{\\text{dec}}\\in\\mathbb R^{m\\times d}$ (so \u0026ldquo;what am I, the output token being written, looking for?\u0026rdquo;), and the key/value sequence is the frozen encoder memory $H_{\\text{enc}}\\in\\mathbb R^{n\\times d}$ (so \u0026ldquo;what does each input token contain, and what information should it contribute?\u0026rdquo;):\n$$ \\boxed{\\ Q = H_{\\text{dec}}\\,W_Q^{c}\\in\\mathbb R^{m\\times d_k},\\qquad K = H_{\\text{enc}}\\,W_K^{c}\\in\\mathbb R^{n\\times d_k},\\qquad V = H_{\\text{enc}}\\,W_V^{c}\\in\\mathbb R^{n\\times d_v}.\\ } $$The projections $W_Q^c,W_K^c,W_V^c$ are [LEARNED], distinct from the decoder\u0026rsquo;s self-attention projections and from the encoder\u0026rsquo;s. Then the operation is identical in form to everything you know:\n$$ \\mathrm{CrossAttn}(H_{\\text{dec}},H_{\\text{enc}})=\\mathrm{softmax}\\!\\Big(\\tfrac{QK^\\top}{\\sqrt{d_k}}+M^{\\text{pad}}\\Big)V . $$That is the whole mechanism. The conceptual weight is entirely in where $Q$ and $K,V$ come from — three matrices, two sources.\nFigure 1. Three matrices, two sources.\n2. The shape that gives it away: a non-square attention matrix In self-attention $Q$ and $K$ have the same number of rows, so the attention matrix is square ($n\\times n$ or $m\\times m$). In cross-attention the query count $m$ (output length) and the key count $n$ (input length) are different, so the score and attention matrices are rectangular:\n$$ S=\\frac{QK^\\top}{\\sqrt{d_k}}\\in\\mathbb R^{m\\times n},\\qquad A=\\mathrm{softmax}(S+M^{\\text{pad}})\\in\\mathbb R^{m\\times n},\\qquad \\mathrm{CrossAttn}=AV\\in\\mathbb R^{m\\times d_v}. $$Read the shapes carefully, because they encode the semantics. Row $i$ of $A$ is a distribution over the $n$ input positions — it is \u0026ldquo;how much output token $i$ attends to each source token.\u0026rdquo; The softmax is taken over the $n$ key positions (each query row sums to 1: $A\\mathbf 1_n=\\mathbf 1_m$). The output $AV\\in\\mathbb R^{m\\times d_v}$ has one vector per output position, formed as a convex combination of the input value vectors:\n$$ \\mathbf o_i=\\sum_{j=1}^{n}A_{ij}\\,\\mathbf v_j,\\qquad A_{ij}=\\frac{\\exp(s_{ij})}{\\sum_{k=1}^{n}\\exp(s_{ik})},\\qquad s_{ij}=\\frac{\\mathbf q_i^\\top\\mathbf k_j}{\\sqrt{d_k}} . $$That rectangular $A$ is a soft alignment between output and input — for translation, $A_{ij}$ tends to light up where output word $i$ corresponds to source word $j$ (Lesson 9 makes this concrete). It is the modern, learned, differentiable replacement for the explicit alignment tables of classical statistical machine translation.\nflowchart LR Hdec[\"H_dec (m × d)output positions\"] --\u003e|\"× W_Q^c\"| Q[\"Q (m × d_k)\"] Henc[\"H_enc (n × d)input memory (frozen)\"] --\u003e|\"× W_K^c\"| K[\"K (n × d_k)\"] Henc --\u003e|\"× W_V^c\"| V[\"V (n × d_v)\"] Q --\u003e S[\"S = QKᵀ/√d_k(m × n) — rectangular!\"] K --\u003e S S --\u003e A[\"A = softmax(S + M_pad)rows over n input positions\"] V --\u003e O[\"O = A·V (m × d_v)one vector per output position\"] A --\u003e O 3. Which masks apply — and which do not This is the subtlety that trips people up, so state it sharply. Cross-attention uses no causal mask. The decoder may attend to any input position, including ones \u0026ldquo;after\u0026rdquo; the current output index — there is no causality between output position $i$ and input position $j$, because they live in different sequences and the entire input is already known. Causality is enforced only in the decoder\u0026rsquo;s self-attention (Lesson 3); cross-attention is deliberately unmasked in that sense.\nWhat cross-attention does use is the encoder\u0026rsquo;s padding mask from Lesson 2, applied on the key axis: output tokens must never attend to \u0026lt;pad\u0026gt; positions in the source.\n$$ M^{\\text{pad}}_{ij}=\\begin{cases}0,\u0026 x_j\\neq\\texttt{\\textless{}pad\\textgreater{}}\\\\ -\\infty,\u0026 x_j=\\texttt{\\textless{}pad\\textgreater{}}\\end{cases}\\quad(\\text{same for every output row }i). $$So the two attention sublayers in a decoder block use different masks: self-attention uses the causal mask (square, lower-triangular), cross-attention uses the source padding mask (rectangular, column-wise). Confusing them is the most common error in implementing a decoder.\nFigure 2. Two sublayers in one block, two different masks.\n4. A tiny worked example Take an output token attending over $n=3$ source tokens, with $d_k=d_v=2$. Suppose the decoder produced query $\\mathbf q$ and the encoder memory produced keys/values:\n$$ \\mathbf q=(1,1),\\quad \\mathbf k_1=(1,1),\\ \\mathbf k_2=(1,0),\\ \\mathbf k_3=(-1,0),\\quad \\mathbf v_1=(1,0),\\ \\mathbf v_2=(0,1),\\ \\mathbf v_3=(1,1). $$Scores (dot product over $\\sqrt{d_k}=\\sqrt2\\approx1.414$):\n$$ s_1=\\tfrac{2}{1.414}=1.414,\\quad s_2=\\tfrac{1}{1.414}=0.707,\\quad s_3=\\tfrac{-1}{1.414}=-0.707 . $$Softmax of $(1.414,\\,0.707,\\,-0.707)$: exponentiate to $(4.11,\\,2.03,\\,0.49)$, sum $=6.63$, giving the alignment weights\n$$ A=(0.620,\\ 0.306,\\ 0.074),\\qquad \\textstyle\\sum_j A_j=1 . $$The output token aligns most strongly to source token 1, partly to token 2, and almost ignores token 3 (whose key points away from the query). Output = weighted blend of the input value vectors:\n$$ \\mathbf o = 0.620\\,(1,0)+0.306\\,(0,1)+0.074\\,(1,1)=(0.694,\\ 0.380). $$Two things to notice. The result is built entirely from encoder values $\\mathbf v_j$ — information flows from the input into the output position. And the weights came from comparing a decoder query against encoder keys — the match between what the output is looking for and what each input token offers. That is conditioning on $x$, made differentiable.\nFigure 3. One output token aligning over three source tokens.\n5. Gradient flow: cross-attention is the bridge for learning, too Cross-attention is not only the forward-pass bridge; it is the only path by which a training gradient reaches the encoder. During the backward pass (Lesson 8), the loss is computed at the decoder\u0026rsquo;s output head. Gradients flow back through the decoder, and at each cross-attention sublayer they split: part updates the decoder\u0026rsquo;s $W_Q^c$ (through $Q=H_{\\text{dec}}W_Q^c$), and part flows through $K=H_{\\text{enc}}W_K^c$ and $V=H_{\\text{enc}}W_V^c$ into $H_{\\text{enc}}$, and from there backward through the entire encoder stack.\n$$ \\frac{\\partial\\mathcal L}{\\partial H_{\\text{enc}}} =\\underbrace{\\frac{\\partial\\mathcal L}{\\partial K}\\frac{\\partial K}{\\partial H_{\\text{enc}}}}_{\\text{via }W_K^c} +\\underbrace{\\frac{\\partial\\mathcal L}{\\partial V}\\frac{\\partial V}{\\partial H_{\\text{enc}}}}_{\\text{via }W_V^c} \\quad\\Longrightarrow\\quad \\text{encoder weights get updated.} $$This is why the two stacks are trained jointly, end to end, even though only the decoder has a loss head: cut the cross-attention connection and the encoder would receive no learning signal at all. The encoder learns to produce representations useful to the decoder\u0026rsquo;s queries precisely because the gradient travels this bridge.\nLearned / config / state Quantity Category Cross-attn projections $W_Q^c,W_K^c,W_V^c,W_O^c$ [LEARNED] $\\sqrt{d_k}$ scaling, \u0026ldquo;no causal mask\u0026rdquo; policy, head count [CONFIG] Source padding mask $M^{\\text{pad}}$ (data-dependent) [CONFIG]/[STATE] Scores $S$, alignment $A\\in\\mathbb R^{m\\times n}$, output $AV$ [STATE] Conclusion Cross-attention is scaled dot-product attention with one change of source: queries from the decoder, keys and values from the encoder memory, $Q=H_{\\text{dec}}W_Q^c$, $K,V=H_{\\text{enc}}W_K^c, H_{\\text{enc}}W_V^c$. The resulting attention matrix is rectangular ($m\\times n$) — a soft, learned alignment whose row $i$ is a distribution over the $n$ input positions, and whose output blends input value vectors into each output position. It uses no causal mask (the input is fully known) but does apply the source padding mask. And it is the sole pathway by which training gradients reach the encoder, which is why the two stacks are learned jointly. Everything about the operation — the softmax, the $\\sqrt{d_k}$, the multi-head structure — is what you already knew; only the two sources are new. This is the mechanism the entire course\u0026rsquo;s invariant names.\nWith all three sublayers now derived, Lesson 5 assembles them into the complete encoder–decoder block, counts the parameters, draws the whole architecture, and pins down where T5 and BART differ from the textbook version.\n← Lesson 3 — The Decoder Stack · Course index · Next: Lesson 5 — The Encoder–Decoder Block → Appendices: A — Formulas · B — Architecture Comparison · C — Acronyms The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/tutorials/encoder-decoder-lessons/lesson-04-cross-attention/","summary":"\u003ch1 id=\"lesson-4--cross-attention-the-only-new-mechanism\"\u003eLesson 4 — Cross-Attention: The Only New Mechanism\u003c/h1\u003e\n\u003cp\u003e\u003cem\u003eTechnical course on encoder–decoder transformers — Lesson 4 of 10.\u003c/em\u003e\u003c/p\u003e\n\u003chr\u003e\n\u003ch2 id=\"recap-of-lesson-3\"\u003eRecap of Lesson 3\u003c/h2\u003e\n\u003cp\u003eThe decoder block has \u003cstrong\u003ethree\u003c/strong\u003e pre-norm residual sublayers: causal self-attention over the output so far, then \u003cstrong\u003ecross-attention\u003c/strong\u003e to the encoder memory, then an FFN. The first and last are inherited unchanged from a decoder-only model; the middle one is the single new operation in the whole architecture, and the decoder carries a second full set of attention projections to run it. We deferred its derivation twice. This lesson pays the debt: cross-attention is scaled dot-product attention with one change — its queries come from the decoder, its keys and values from the encoder.\u003c/p\u003e","title":"Cross-Attention: The Only New Mechanism"},{"content":"Lesson 4 — Sparse Top-k Computation \u0026amp; the Combine Technical course on MoE LLM mechanics — Lesson 4 of 10.\nRecap of Lesson 3 The router is a learned matrix $W_g\\in\\mathbb R^{E\\times d}$ producing logits $\\mathbf h=W_g\\mathbf x$, a softmax distribution $\\mathbf P=\\mathrm{softmax}(\\mathbf h)$ over experts, and a top-$k$ selection $\\mathcal T_k(\\mathbf x)$ whose renormalized probabilities are the gate weights $g_i$. The selection is discrete ($\\arg\\text{top-}k$ has zero gradient), so the router can be trained only through the continuous gate weights of the experts it chose. This lesson assembles the complete sparse forward pass, then derives exactly what gradient flows back — to the experts and to the router — making \u0026ldquo;trained through the weights, not the choice\u0026rdquo; precise.\nWe now have all the pieces: experts (Lesson 2) and the router that picks among them (Lesson 3). Putting them together raises two questions. Forward: what does the layer actually compute, and in what sense is it \u0026ldquo;sparse\u0026rdquo;? Backward: how does gradient descent improve a layer that made a hard choice? This lesson answers both.\n1. The forward pass, end to end For a single token $\\mathbf x$, the MoE layer executes:\nRoute: $\\mathbf h=W_g\\mathbf x$; pick $\\mathcal T_k(\\mathbf x)=\\arg\\text{top-}k_i\\,h_i$; form gate weights $g_i=\\dfrac{e^{h_i}}{\\sum_{j\\in\\mathcal T_k}e^{h_j}}$ for $i\\in\\mathcal T_k$. Compute experts: evaluate $E_i(\\mathbf x)$ only for $i\\in\\mathcal T_k(\\mathbf x)$. Combine: $\\displaystyle \\mathbf y=\\mathrm{MoE}(\\mathbf x)=\\sum_{i\\in\\mathcal T_k(\\mathbf x)} g_i(\\mathbf x)\\,E_i(\\mathbf x).$ Steps 2–3 are the combine: a convex (for $k\\gt 1$) weighted sum of the selected experts\u0026rsquo; outputs, each of dimension $d$, producing a $d$-vector that re-enters the residual stream. Note that the $E-k$ unselected experts are never evaluated for this token — the layer computes a token-specific sub-network on the fly.\nflowchart LR x[\"token x ∈ ℝᵈ\"] --\u003e R[\"routerh = W_g x\"] R --\u003e S[\"top-k select𝒯ₖ, gate weights gᵢ\"] S --\u003e|\"i ∈ 𝒯ₖ only\"| E1[\"Expert E_a\"] S --\u003e|\"i ∈ 𝒯ₖ only\"| E2[\"Expert E_b\"] E1 --\u003e C[\"combiney = Σ gᵢ Eᵢ(x)\"] E2 --\u003e C C --\u003e out[\"y ∈ ℝᵈ → residual\"] Figure 1. Route, compute, combine — the layer\u0026rsquo;s three steps.\n2. Sparse in compute, dense in memory This single layer crystallizes the Lesson 1 asymmetry.\nSparse in compute. Only $k$ of $E$ experts run, so the FLOPs are those of $k$ FFNs (plus the negligible router). Per token, expert FLOPs are $\\approx 2\\cdot k\\cdot 2dd_{ff}=8kd^2$ — independent of $E$. Raise $E$ to $1000$ and the forward cost of this layer does not move.\nDense in memory. All $E$ experts must be resident, because the router is free to send some token in the batch to any expert. You cannot keep only the \u0026ldquo;popular\u0026rdquo; experts loaded — routing is data-dependent and decided at runtime. So the layer\u0026rsquo;s memory footprint is all $E$ experts, $\\approx 8Ed^2$, regardless of $k$.\n$$ \\text{compute} \\propto k \\quad\\text{(activated)},\\qquad \\text{memory}\\propto E\\quad\\text{(total)}. $$This is the same \u0026ldquo;cheap FLOPs, expensive bytes\u0026rdquo; signature from Lesson 1, now visible at the level of a single layer. Every systems consequence in Lessons 6, 7, and 10 is a way of managing the fact that compute and storage scale with different numbers.\nFigure 2. One layer, two scalings.\n3. The backward pass — gradient to the experts The combine is an ordinary differentiable function of the expert outputs, so the gradient to a selected expert is unsurprising. With upstream gradient $\\boldsymbol\\delta=\\partial\\mathcal L/\\partial\\mathbf y$, for each $i\\in\\mathcal T_k$:\n$$ \\frac{\\partial\\mathcal L}{\\partial E_i(\\mathbf x)}=g_i(\\mathbf x)\\,\\boldsymbol\\delta, $$which then flows into $E_i$\u0026rsquo;s own weights by the standard FFN backward pass (companion course, Lesson 7). The gate weight $g_i$ simply scales the gradient each expert receives — an expert the router trusted more for this token gets a proportionally larger update from it. The $E-k$ unselected experts receive zero gradient from this token: they were not in the forward computation, so by the chain rule no gradient returns to them. An expert only learns from the tokens routed to it. This is the seed of the dead-expert problem (Lesson 5): an expert that stops being chosen stops being trained, and so never improves enough to be chosen again.\n4. The backward pass — gradient to the router Here is the subtle part promised in Lesson 3. The output depends on the router logits $\\mathbf h$ in two ways: through the selection $\\mathcal T_k$ (discrete) and through the gate weights $g_i$ (continuous). The selection contributes no gradient ($\\partial\\mathcal T_k/\\partial\\mathbf h=0$ almost everywhere), so the only path back to $W_g$ is through the $g_i$:\n$$ \\frac{\\partial\\mathcal L}{\\partial g_i}=\\boldsymbol\\delta^\\top E_i(\\mathbf x)\\quad (i\\in\\mathcal T_k),\\qquad \\frac{\\partial\\mathcal L}{\\partial h_j}=\\sum_{i\\in\\mathcal T_k}\\frac{\\partial\\mathcal L}{\\partial g_i}\\frac{\\partial g_i}{\\partial h_j}, $$and since $g_i$ is a softmax over the selected logits, $\\partial g_i/\\partial h_j$ is the familiar softmax Jacobian restricted to $\\mathcal T_k$:\n$$ \\frac{\\partial g_i}{\\partial h_j}=g_i(\\delta_{ij}-g_j),\\qquad i,j\\in\\mathcal T_k. $$Reading this off: the router receives gradient only on the logits of the experts it selected, and that gradient says \u0026ldquo;increase the weight on the selected experts whose outputs reduced the loss, decrease it on those that raised it.\u0026rdquo; It is a relative reweighting among the winners. What the router never receives is a term telling it that an unselected expert would have done better — that information is invisible because the unselected experts were never computed. (This is exactly why $k=1$ Switch keeps the chosen expert\u0026rsquo;s probability as a literal multiplier $g=p_{\\text{chosen}}$ on its output: without it, a $k=1$ gate weight would be the constant $1$, its softmax Jacobian would vanish, and the router would get no gradient at all.)\nContrast with the dense model. In a dense FFN, every parameter is on the path for every token, so the backward pass reaches all of $\\theta$ on every example — gradient coverage is total and automatic. In an MoE layer the backward pass reaches only the selected experts and only the selected router logits. Coverage is partial and self-reinforcing: experts that are chosen get trained and get better and get chosen more; experts that fall out of favor get no gradient and decay into uselessness. The smooth, egalitarian gradient flow of a dense model is replaced by a sparse, winner-take-most flow — and correcting that bias is the subject of the next lesson.\nFigure 3. Three destinations for the gradient, and one of them gets nothing.\n5. Learned / config / state Quantity Category Expert weights (updated for selected experts only) [LEARNED] Router $W_g$ (updated on selected logits only) [LEARNED] $k$, softmax placement, combine form (with/without shared experts) [CONFIG] Selection $\\mathcal T_k$, gate weights $g_i$, expert outputs $E_i(\\mathbf x)$, upstream $\\boldsymbol\\delta$ [STATE] Conclusion The MoE forward pass routes ($\\mathbf h=W_g\\mathbf x$, top-$k$, gate weights $g_i$), computes only the $k$ selected experts, and combines them as $\\mathbf y=\\sum_{i\\in\\mathcal T_k}g_iE_i(\\mathbf x)$ — sparse in compute (FLOPs $\\propto k$, independent of $E$), dense in memory (all $E$ experts resident). Backward, each selected expert receives $g_i\\boldsymbol\\delta$ (gate-scaled) and unselected experts receive nothing; the router receives gradient only on its selected logits, through the softmax Jacobian $g_i(\\delta_{ij}-g_j)$ — a relative reweighting of the winners, never a signal to choose differently. This partial, self-reinforcing gradient coverage is the structural reason an MoE can develop dead experts, the failure the next lesson confronts head-on.\nWe have a layer that computes and trains correctly per token — but \u0026ldquo;correct per token\u0026rdquo; is exactly what allows the router to quietly funnel everything to a few favorites. Lesson 5 measures that imbalance and adds the explicit pressure that fixes it.\n← Lesson 3 — The Router / Gating Network · Course index · Next: Lesson 5 — Load Balancing \u0026amp; Auxiliary Losses → Appendices: A — MoE Family · B — Dense vs. MoE · C — Acronyms The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/tutorials/moe-lessons/lesson-04-sparse-computation-and-combine/","summary":"\u003ch1 id=\"lesson-4--sparse-top-k-computation--the-combine\"\u003eLesson 4 — Sparse Top-k Computation \u0026amp; the Combine\u003c/h1\u003e\n\u003cp\u003e\u003cem\u003eTechnical course on MoE LLM mechanics — Lesson 4 of 10.\u003c/em\u003e\u003c/p\u003e\n\u003chr\u003e\n\u003ch2 id=\"recap-of-lesson-3\"\u003eRecap of Lesson 3\u003c/h2\u003e\n\u003cp\u003eThe router is a learned matrix $W_g\\in\\mathbb R^{E\\times d}$ producing logits $\\mathbf h=W_g\\mathbf x$, a softmax distribution $\\mathbf P=\\mathrm{softmax}(\\mathbf h)$ over experts, and a top-$k$ selection $\\mathcal T_k(\\mathbf x)$ whose renormalized probabilities are the gate weights $g_i$. The selection is discrete ($\\arg\\text{top-}k$ has zero gradient), so the router can be trained only through the continuous gate weights of the experts it chose. This lesson assembles the complete sparse forward pass, then derives exactly what gradient flows back — to the experts and to the router — making \u0026ldquo;trained through the weights, not the choice\u0026rdquo; precise.\u003c/p\u003e","title":"Sparse Top-k Computation and the Combine"},{"content":"Lesson 4 — The Encoder Block \u0026amp; Stack Technical course on encoder LLM mechanics — Lesson 4 of 10.\nRecap of Lesson 3 Bidirectional self-attention is scaled dot-product attention with the causal mask deleted ($M_{ij}=0$ over real tokens), so the attention matrix is full and every token\u0026rsquo;s vector depends on the whole sequence. This makes the model an encoder, makes next-token training impossible, and removes the KV cache and any ability to generate; the only surviving mask is the padding mask. We now wrap this attention operator into a full block and stack $N$ of them, and ask what the resulting representations are.\n1. The block: same parts, one notable wiring choice An encoder block has the same two sub-layers as a decoder block: a multi-head self-attention sub-layer and a position-wise MLP (feed-forward network), each wrapped in a residual connection and LayerNorm. For layer $\\ell$, the original BERT computes\n$$ \\begin{aligned} Z \u0026= \\mathrm{LayerNorm}\\big(H^{(\\ell-1)} + \\mathrm{MHA}(H^{(\\ell-1)})\\big),\\\\[2pt] H^{(\\ell)} \u0026= \\mathrm{LayerNorm}\\big(Z + \\mathrm{MLP}(Z)\\big), \\end{aligned} $$where the MLP is the familiar two-layer expansion with a GELU nonlinearity,\n$$ \\mathrm{MLP}(\\mathbf z)=W_2\\,\\mathrm{GELU}(W_1\\mathbf z+\\mathbf b_1)+\\mathbf b_2,\\qquad W_1\\in\\mathbb R^{d_{ff}\\times d},\\ W_2\\in\\mathbb R^{d\\times d_{ff}}, $$with the standard $d_{ff}=4d$ [CONFIG]. All of $W_Q,W_K,W_V,W_O,W_1,W_2$ and the LayerNorm gains/shifts are [LEARNED], exactly as in the decoder. The parameter budget per block is dominated by the MLP ($2\\cdot4d^2=8d^2$) plus attention ($4d^2$), about $12d^2$ per layer — identical accounting to the decoder.\nPost-LN vs. pre-LN The one wiring detail worth flagging: the equations above place LayerNorm after the residual addition (post-LN), which is the original BERT/Transformer arrangement. Most modern decoders use pre-LN, normalizing inside the residual branch:\n$$ H^{(\\ell)} = H^{(\\ell-1)} + \\mathrm{MHA}\\big(\\mathrm{LayerNorm}(H^{(\\ell-1)})\\big),\\quad\\text{etc.} $$Pre-LN gives a cleaner gradient path (the residual stream is never renormalized), which is why it became the default for very deep, hard-to-train decoders; post-LN can match or slightly beat it at BERT\u0026rsquo;s modest depth but needs careful learning-rate warmup to stay stable. This is not an encoder-vs-decoder distinction in principle — it is a historical-default distinction. ModernBERT (Appendix A) switches encoders to pre-LN. Either way, the residual structure is what lets gradients flow through $N$ layers without vanishing, exactly as derived in the decoder course.\nContrast with the decoder. The block math is essentially the same; only two cosmetic-but-real differences recur: original encoders use post-LN (decoders trend pre-LN), and encoders historically use GELU while many decoders moved to SwiGLU. The attention sub-layer, of course, carries the bidirectional (unmasked) attention of Lesson 3 — that is the substantive difference, and it lives entirely inside MHA.\nFigure 1. Post-LN and pre-LN: where the normalization sits.\n2. The stack and the residual stream Stack $N$ blocks: $H^{(0)}\\to H^{(1)}\\to\\cdots\\to H^{(N)}$, each $H^{(\\ell)}\\in\\mathbb R^{n\\times d}$. BERT-base uses $N=12$, $d=768$, $h=12$ (so $d_k=64$); BERT-large uses $N=24$, $d=1024$, $h=16$. The running tensor $H^{(\\ell)}$ is the residual stream: a width-$d$ vector per token that each block reads from and writes back to additively.\nThe key conceptual point, and the reason an encoder is an encoder all the way up: at every layer the shape is $n\\times d$ — one vector per input token — and at no layer does the model collapse to a single prediction. The decoder, too, keeps an $n\\times d$ stream internally, but it exists to feed a next-token head at the top. The encoder\u0026rsquo;s stream exists to be the output. The final $H^{(N)}$ is precisely the matrix of contextual representations promised in Lesson 1\u0026rsquo;s definition $g(x;\\theta)=H$.\n3. What \u0026ldquo;contextual embedding\u0026rdquo; means, precisely Lesson 2 gave each token a static input embedding $E_{\\text{tok}}[x_i]$ — the same vector for \u0026ldquo;bank\u0026rdquo; regardless of context. After the stack, the output row $H^{(N)}_{i,:}$ is a contextual embedding: a function of the whole sequence, so \u0026ldquo;bank\u0026rdquo; near \u0026ldquo;river\u0026rdquo; and \u0026ldquo;bank\u0026rdquo; near \u0026ldquo;money\u0026rdquo; land at different points in $\\mathbb R^{d}$.\nWe can be quantitative about where the context enters. The only sub-layer that mixes information across positions is attention; the MLP and LayerNorm act on each position independently. So all contextualization is the accumulated effect of $N$ rounds of bidirectional mixing. After one layer, $H^{(1)}_{i}$ depends on every input token directly (full attention). After two, it depends on every token\u0026rsquo;s layer-1 representation, which already depended on everything — and so on. Depth deepens the composition of context, not its reach (the reach is already global after layer 1). This is unlike a convolutional or windowed model where reach grows with depth; in a full-attention encoder, reach is immediate and depth buys abstraction.\nProbing intuition. Empirically, lower layers of a BERT stack encode more surface/syntactic features (morphology, part of speech) and upper layers encode more semantic/task-relevant features — a rough \u0026ldquo;syntax low, semantics high\u0026rdquo; gradient. You do not need to take this as doctrine, but it explains why some fine-tuning recipes read from intermediate layers rather than only $H^{(N)}$.\nFigure 2. Reach is immediate; depth buys abstraction.\n4. The [CLS] register: a sequence summary built by attention Recall [CLS] sits at position 0 of every input (Lesson 2). It carries no word meaning of its own, which is the point: it is a blank slot whose output vector $H^{(N)}_{0,:}$ the model is free to fill with a summary of the whole sequence. Because attention is bidirectional, the [CLS] position attends to all real tokens at every layer, so by the top of the stack its vector is a learned, weighted aggregation of the entire input.\nNothing forces [CLS] to be a good summary in the raw pretrained model — it becomes one because the pretraining objective and (especially) fine-tuning push gradient signal through it. In BERT\u0026rsquo;s original pretraining, the Next-Sentence-Prediction head (Lesson 5) reads from [CLS], giving it an early reason to aggregate. In fine-tuning for classification (Lesson 9), the classifier reads from [CLS], and gradients then sculpt it into exactly the summary that task needs.\nflowchart LR I[\"[CLS] w₁ w₂ … w_k [SEP]\"] --\u003e S[\"N bidirectional blocks\"] S --\u003e O[\"H^(N): one vector per token\"] O --\u003e P[\"H^(N)[0] = [CLS] vector→ sequence-level head\"] O --\u003e Q[\"H^(N)[i] = token i vector→ token-level head\"] Contrast with the decoder. A decoder has a privileged position too — the last token, whose top-layer vector feeds the next-token head, because causal masking means only the last position has seen the whole prefix. An encoder\u0026rsquo;s privileged position is the first token ([CLS]), and it is privileged by convention plus bidirectional attention, not by masking: every position has seen everything, so any position could serve as the read-out, but we designate position 0 and train it to be the summary.\nFigure 3. How position 0 becomes a whole-sequence summary.\n5. Where this leaves us The stack turns $H^{(0)}$ into $H^{(N)}\\in\\mathbb R^{n\\times d}$: contextual vectors for every token, plus a [CLS] vector primed to summarize the whole input. That matrix is the encoder\u0026rsquo;s entire output — there are no logits yet, no probabilities, no prediction. To get predictions we attach heads (Lesson 6), and to train those representations to be useful in the first place we need the objective that gives the stack its learning signal. That objective is the subject of Lesson 5, and it is where the encoder departs most sharply from everything you know about decoders.\nConclusion An encoder block is the standard transformer block — multi-head attention + MLP, each in a residual with LayerNorm — carrying the bidirectional (unmasked) attention of Lesson 3. Original BERT wires it post-LN with GELU; the choices are historical defaults, not deep necessities (ModernBERT moves to pre-LN). Stacking $N$ blocks maintains an $n\\times d$ residual stream at every layer and never collapses to a prediction: the final $H^{(N)}$ is the matrix of contextual embeddings that defines the encoder. Context enters only through attention, is global after a single layer, and is deepened (not widened) by depth. The [CLS] position, attended-to by everything and read-from by sequence-level heads, becomes a learned whole-sequence summary — the encoder\u0026rsquo;s analogue of the decoder\u0026rsquo;s privileged last token, but privileged by convention and bidirectional reach rather than by causal masking.\nNext: the objective. Lesson 5 derives masked language modeling — why bidirectionality forces a denoising loss, how the 80/10/10 corruption works, and what Next-Sentence Prediction added (and why it was later dropped).\n← Previous: Lesson 3 — Bidirectional Self-Attention · Course index · Next: Lesson 5 — Masked Language Modeling → Appendices: A — Encoder Family · B — Encoder vs. Decoder Cheat Sheet · C — Acronyms The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/tutorials/encoder-deep-dive/lesson-04-the-encoder-block-and-stack/","summary":"\u003ch1 id=\"lesson-4--the-encoder-block--stack\"\u003eLesson 4 — The Encoder Block \u0026amp; Stack\u003c/h1\u003e\n\u003cp\u003e\u003cem\u003eTechnical course on encoder LLM mechanics — Lesson 4 of 10.\u003c/em\u003e\u003c/p\u003e\n\u003chr\u003e\n\u003ch2 id=\"recap-of-lesson-3\"\u003eRecap of Lesson 3\u003c/h2\u003e\n\u003cp\u003eBidirectional self-attention is scaled dot-product attention with the causal mask deleted ($M_{ij}=0$ over real tokens), so the attention matrix is full and every token\u0026rsquo;s vector depends on the whole sequence. This makes the model an encoder, makes next-token training impossible, and removes the KV cache and any ability to generate; the only surviving mask is the padding mask. We now wrap this attention operator into a full block and stack $N$ of them, and ask what the resulting representations \u003cem\u003eare\u003c/em\u003e.\u003c/p\u003e","title":"The Encoder Block and Stack"},{"content":"Chapter 4 — The Storytellers: Decoder Models We now have our three foundations in hand: a model is a function set by learned knobs, text becomes points on a map of meaning, and attention lets those points talk to one another. From here on, we are simply arranging this machinery into different shapes for different jobs. The first and most famous shape is the decoder model — the kind of system behind ChatGPT, Claude, Llama, and Gemini. These are the storytellers: models built to write, one word at a time. This chapter is about how they work and what makes them so capable. (We will save the deep details of how they are trained and run for Part IV, where those topics get a full treatment across all model types. Here we focus on the architecture itself.)\nA storyteller writes left to right The defining trait of a decoder model is that it generates text the way you write a sentence: from left to right, one word at a time, each new word committed before the next is chosen. When the model is choosing its next word, it is allowed to look at everything it has written so far — but never at anything that comes after, because nothing after it exists yet. It is composing into the void.\nThis left-to-right rule has a name — causal attention — and a simple justification. Recall that these models are trained on the game of guessing the next word. That game only teaches anything if the model genuinely cannot see the answer. If, while guessing the word that comes next, it were allowed to peek at the word that actually comes next, the guess would be trivial cheating and the model would learn nothing. So during both training and use, each word\u0026rsquo;s attention is restricted to look only backward, at what has already been written. The storyteller writes blind to the future, exactly as a human author does.\nFigure 1. Blinders on the road ahead.\nHow a tower of blocks refines meaning A decoder model is not a single layer of attention. It is the same building block, stacked dozens of times — sometimes more than a hundred — into a tall tower. Your text enters at the bottom and rises through the stack, getting a little more refined and a little more context-aware at each level. Understanding what one of these blocks does, and why stacking them works, is most of understanding the whole machine.\nEach block does two things in turn. First comes attention — the conversation among words from Chapter 3, where each word looks back over the text and pulls in what is relevant to it. This is the mixing step, where information flows sideways between words. Then comes a second stage in which each word, now holding the context it gathered, is processed on its own — transformed, refined, reconsidered in place. This second stage does not look at other words at all; it works on each word individually, and, perhaps surprisingly, it is where most of the model\u0026rsquo;s raw stored knowledge actually lives.\nA clean way to picture the rhythm: attention is the group discussion where everyone shares notes, and the second stage is each person going off alone to think hard about what they just heard. Discussion, private reflection, discussion, private reflection — over and over, all the way up the tower. By the time a word reaches the top, it has been through dozens of rounds of gathering context and digesting it.\nResearchers find that the layers loosely divide the labor among themselves. The early layers tend to handle surface features — basic grammar, word forms. The middle layers handle meaning and relationships. The late layers assemble the final answer. But, as always, nobody designs this division. It emerges on its own because it is a useful way to organize the work.\nTwo pieces of plumbing that make tall towers possible Stacking dozens of layers sounds straightforward, but for years it was not. Deep towers tended to fail to learn, because the faint signal that tells each layer how to improve would fade to nothing as it traveled down through so many layers. Two unglamorous but essential pieces of plumbing fixed this, and they appear in every model in this book.\nThe first is the shortcut connection. Rather than each block replacing the information it received, it adds its contribution on top of what came in. This creates a clean highway running straight up through the entire tower, so information and the learning signal can travel the full height without getting lost. This single idea — let each layer add to a running stream rather than overwrite it — is what made very deep networks trainable at all. It is hard to overstate how important it turned out to be.\nThe second is normalization. As information climbs the tower, its raw scale can drift — growing too large or shrinking too small — and a deep stack of such drifts can spiral out of control. Normalization gently rescales the numbers at each step to keep them in a sane range, like a volume governor on a sound system that prevents runaway feedback. It keeps the deep tower stable enough to train and to run.\nFigure 2. The tower of blocks, and the plumbing that keeps it standing.\nTurning the top of the tower into a guess After a word\u0026rsquo;s representation has risen through every block, it is rich, fully contextual, and ready — but it is still just an internal bundle of numbers, not yet a prediction. The final step, the output head, translates that bundle into a score for every possible next token in the model\u0026rsquo;s vocabulary. A high score means \u0026ldquo;this token is a likely continuation.\u0026rdquo;\nThose raw scores are then passed through a step that converts them into clean probabilities adding up to one hundred percent. This step has a useful exaggerating quality: a token scoring a little higher than the rest ends up with a lot more of the probability. The model does not hedge limply across thousands of equally weak options; it commits, concentrating its confidence on a handful of plausible continuations. The result is a ranked set of odds: perhaps seventy percent on one word, eight percent on another, and a long thin tail of unlikely possibilities.\nNotice what the model produces: not a word, but a distribution over possible words. Which word actually gets chosen from that distribution is a separate decision — and a surprisingly consequential one, full of dials that change the model\u0026rsquo;s whole personality. That choosing step is where a great deal of the texture of these systems lives, and we devote much of Chapter 11 to it.\nWriting by repetition Now put the pieces together to see how a storyteller actually produces a paragraph. You give it some text. It runs that text up through the tower and produces its odds for the next word. One word is chosen and glued onto the end. Then the whole, now-slightly-longer text is run through again, producing odds for the next next word, which is chosen and glued on. And again, and again. The essay, the answer, the poem, the working program — all of it is built one word at a time by this patient loop of predict, choose, append, repeat.\nThis is worth pausing on, because it is genuinely the trick. There is no grand plan formed in advance, no outline the model fills in. Coherence over a long passage emerges from the fact that each new word is chosen in full view of everything written so far. The model writes its way forward, and the structure holds because every step is conditioned on the whole growing context. A storyteller that has merely learned to choose each next word well, given everything before it, turns out to be able to write things that look very much like thought.\nFigure 3. Predict, choose, append, repeat.\nWhat to carry forward A decoder model is a left-to-right storyteller. It writes one word at a time, each word allowed to attend only to what came before, because its whole education was the game of guessing the next word without peeking. Inside, your text rises through a tall tower of identical blocks, each block letting words gather context through attention and then digest it privately — kept stable and trainable by shortcut connections and normalization. At the top, an output head turns the result into odds over the next word, one word is chosen, and the loop runs again. From that humble repetition comes the full range of what these systems can do.\nThe decoder is built to write. But not every job is a writing job. Sometimes you do not want to generate text at all — you want to deeply understand a piece of text you already have. For that, the same machinery is arranged into a very different shape: a reader rather than a writer. That is Chapter 5 .\n← Chapter 3 · Attention · Contents · Chapter 5 · The Readers → The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/tutorials/inside-the-machines/chapter-04-decoder-models/","summary":"\u003ch1 id=\"chapter-4--the-storytellers-decoder-models\"\u003eChapter 4 — The Storytellers: Decoder Models\u003c/h1\u003e\n\u003cp\u003eWe now have our three foundations in hand: a model is a function set by learned knobs, text becomes points on a map of meaning, and attention lets those points talk to one another. From here on, we are simply arranging this machinery into different shapes for different jobs. The first and most famous shape is the \u003cem\u003edecoder model\u003c/em\u003e — the kind of system behind ChatGPT, Claude, Llama, and Gemini. These are the storytellers: models built to write, one word at a time. This chapter is about how they work and what makes them so capable. (We will save the deep details of how they are \u003cem\u003etrained\u003c/em\u003e and \u003cem\u003erun\u003c/em\u003e for Part IV, where those topics get a full treatment across all model types. Here we focus on the architecture itself.)\u003c/p\u003e","title":"The Storytellers: Decoder Models"},{"content":"Lesson 4 — The Transformer Block Technical course on LLM mechanics — Lesson 4 of 10.\nRecap of Lesson 3 Self-attention computes $A V$ with $A=\\mathrm{softmax}(QK^\\top/\\sqrt{d_k}+M)$, a row-stochastic matrix of learned asymmetric similarities; the $\\sqrt{d_k}$ divisor keeps softmax off its zero-gradient saturation (Jacobian $\\mathrm{diag}(\\mathbf a)-\\mathbf a\\mathbf a^\\top$); the causal mask enforces autoregression. Multi-head attention uses $h$ subspaces ($d_k=d/h$), costs $\\approx4d^2$ params/layer and $O(n^2d)$ compute. The attention matrix is [STATE]. We now assemble the full block and analyze depth, normalization, and the parameter budget.\nAttention let tokens exchange information, but on its own it is a single layer of mixing — and, as we will see, a stack of them would collapse without help. To build the deep, trainable body of $f$ we need three more ingredients around attention: a per-token nonlinear transform, a way to keep gradients alive through dozens of layers, and a way to keep the activations well-scaled. This lesson adds those three and counts the cost.\n1. Block structure (pre-norm) A modern transformer block composes two residual sub-layers. The pre-norm formulation (normalize before each sub-layer), which is standard because it trains stably at depth:\n$$ \\begin{aligned} \\tilde H \u0026= H + \\mathrm{MHA}\\big(\\mathrm{Norm}(H)\\big),\\\\ H' \u0026= \\tilde H + \\mathrm{MLP}\\big(\\mathrm{Norm}(\\tilde H)\\big). \\end{aligned} $$The body of $f$ is $N$ such blocks, each with its own independent learned weights, $H^{(0)}\\!\\to\\!H^{(1)}\\!\\to\\cdots\\to H^{(N)}$; the depth $N$ is a chosen hyperparameter (the summary table at the end of the lesson catalogues which quantities are learned, config, or state). We examine the three new components in turn: the MLP, the residual, and the norm — first what each computes in the forward pass, then why it earns its place once gradients have to flow backward.\nFigure 1. The pre-norm block: two residual sub-layers.\n2. The position-wise MLP (feed-forward network) Applied identically and independently to each row (token) of its input:\n$$ \\mathrm{MLP}(\\mathbf x)=W_2\\,\\phi\\!\\big(W_1\\mathbf x+\\mathbf b_1\\big)+\\mathbf b_2,\\qquad W_1\\in\\mathbb R^{d_{ff}\\times d},\\ W_2\\in\\mathbb R^{d\\times d_{ff}}. $$The hidden width is typically $d_{ff}=4d$ ([CONFIG]). The up-projection lifts to a higher-dimensional feature space, the nonlinearity gates it, the down-projection compresses back. Where attention mixes information across tokens, the MLP transforms each token in place — the two sub-layers are complementary, and the next subsection shows the nonlinearity is not optional decoration but the thing that makes the whole stack more than a single matrix.\nWhy a nonlinearity is mathematically required Without $\\phi$, $\\mathrm{MLP}(\\mathbf x)=W_2W_1\\mathbf x+\\text{const}=(W_2W_1)\\mathbf x+\\text{const}$, an affine map. Composition of affine maps is affine, so a network of any depth would collapse to a single affine transform — incapable of representing the nonlinear conditional $P_\\theta(x_t\\mid x_{\\lt t})$. The nonlinearity is what grants universal-approximation capacity. Common choices:\nReLU: $\\phi(z)=\\max(0,z)$ — historical baseline. GELU: $\\phi(z)=z\\,\\Phi(z)$, where $\\Phi$ is the standard-normal CDF; a smooth gate, dominant in GPT-family models. SwiGLU (a gated linear unit, used in Llama/PaLM): $\\mathrm{MLP}(\\mathbf x)=\\big(\\mathrm{SiLU}(W_a\\mathbf x)\\odot(W_b\\mathbf x)\\big)W_2$ with $\\mathrm{SiLU}(z)=z\\,\\sigma(z)$. It uses three matrices, so to match parameter count the hidden width is reduced to $\\approx\\tfrac{2}{3}\\cdot4d$. Parameter budget — where the model\u0026rsquo;s mass lives A standard (ungated) MLP has $2\\,d\\,d_{ff}=8d^2$ parameters (with $d_{ff}=4d$). Versus attention\u0026rsquo;s $4d^2$. So per block the MLP is $\\approx\\tfrac{8d^2}{8d^2+4d^2}=\\tfrac{2}{3}$ of parameters. Across the model the embedding/unembedding ($Vd$ each) and the $N$ blocks ($12d^2$ each) give the well-known estimate $P\\approx 12Nd^2 + 2Vd$, dominated by $12Nd^2$ for deep models. Most parameters — and thus most stored knowledge — sit in the MLPs, not attention.\nFigure 2. Where the model\u0026rsquo;s mass lives.\n3. Residual connections and gradient flow We now have the two sub-layers; stacking many of them is what gives the model its capacity. But naively stacking nonlinear transforms is exactly where deep networks historically broke — gradients vanish on the way down. The residual connection is the structural fix, and its value shows up entirely in the backward pass.\nEach sub-layer outputs $H+\\mathcal F(\\mathrm{Norm}(H))$ rather than $\\mathcal F(\\cdot)$ alone. The mathematical payoff is in the backward pass. Differentiating a residual map $\\mathbf y=\\mathbf x+\\mathcal F(\\mathbf x)$:\n$$ \\frac{\\partial \\mathbf y}{\\partial \\mathbf x}=I+\\frac{\\partial\\mathcal F}{\\partial\\mathbf x}. $$Stacking $N$ residual blocks, the input-to-output Jacobian is a product $\\prod_\\ell\\big(I+\\partial\\mathcal F_\\ell\\big)$, which expands to the identity plus many lower-order terms. The identity path guarantees the gradient reaches every layer undiminished, defeating the multiplicative shrink-to-zero that plagues plain deep stacks:\n$$ \\Big\\|\\prod_\\ell \\tfrac{\\partial\\mathcal F_\\ell}{\\partial\\mathbf x}\\Big\\|\\ \\text{can} \\to 0,\\quad\\text{but}\\quad \\prod_\\ell\\big(I+\\tfrac{\\partial\\mathcal F_\\ell}{\\partial\\mathbf x}\\big)\\supseteq I. $$This is what makes 32-, 80-, or 100+-layer training feasible. Residuals add no parameters ([CONFIG] structure). They also impose a convention: every sub-layer must preserve width $d$, so the representation flows along a single \u0026ldquo;residual stream\u0026rdquo; that each sub-layer reads from and writes to additively.\n4. Normalization: forward and backward Residuals keep gradients flowing, but they also let activations accumulate scale as the residual stream is written to layer after layer. Normalization is the counterweight: it rescales each token\u0026rsquo;s representation before every sub-layer reads it, keeping the inputs in a stable range so the whole deep stack trains without the activations drifting.\nLayerNorm Per token vector $\\mathbf x\\in\\mathbb R^d$, normalize across features then apply a learned affine:\n$$ \\mu=\\tfrac1d\\textstyle\\sum_k x_k,\\quad \\sigma^2=\\tfrac1d\\sum_k(x_k-\\mu)^2,\\quad \\mathrm{LN}(\\mathbf x)=\\gamma\\odot\\frac{\\mathbf x-\\mu}{\\sqrt{\\sigma^2+\\epsilon}}+\\beta, $$with a learned gain and shift $\\gamma,\\beta\\in\\mathbb R^d$ and a small fixed constant $\\epsilon\\sim10^{-5}$. Statistics are computed per token over its own $d$ features — no mixing across tokens or batch, which is why LayerNorm (unlike BatchNorm) behaves identically in training and inference and is insensitive to batch size.\nThe backward pass is non-trivial because $\\mu,\\sigma$ depend on all coordinates. Writing $\\hat{\\mathbf x}=(\\mathbf x-\\mu)/\\sqrt{\\sigma^2+\\epsilon}$, the gradient w.r.t. the input is\n$$ \\frac{\\partial\\mathcal L}{\\partial x_i}=\\frac{1}{\\sqrt{\\sigma^2+\\epsilon}}\\Big(g_i-\\tfrac1d\\textstyle\\sum_k g_k-\\hat x_i\\cdot\\tfrac1d\\sum_k g_k\\hat x_k\\Big),\\quad g_i:=\\frac{\\partial\\mathcal L}{\\partial\\hat x_i}=\\gamma_i\\frac{\\partial\\mathcal L}{\\partial \\mathrm{LN}(\\mathbf x)_i}. $$The two subtracted terms project out the components of the upstream gradient along the mean and the normalized direction — normalization removes scale and shift information, so no gradient flows back along those directions.\nRMSNorm Most recent models replace LayerNorm with RMSNorm, which drops mean-centering and the shift:\n$$ \\mathrm{RMS}(\\mathbf x)=\\frac{\\mathbf x}{\\sqrt{\\tfrac1d\\sum_k x_k^2+\\epsilon}}\\odot\\gamma. $$It removes the mean and $\\beta$ computations (fewer operations and one fewer parameter vector) with no measurable quality loss — empirically the re-centering was the inessential part. $\\gamma$ is [LEARNED].\nPre-norm vs. post-norm. The original transformer placed the norm after the residual add (post-norm), which needs careful learning-rate warmup to avoid divergence at depth. Pre-norm (shown above) keeps a clean identity residual path and trains stably without delicate tuning, so it is now standard. This placement is a [CONFIG] choice with real stability consequences.\nFigure 3. LayerNorm, and what RMSNorm drops.\n5. Stacking and emergent layer roles With MLP, residual, and norm in hand, the block is complete; the body of $f$ is just this block repeated. The question that remains is what depth buys you — and the answer turns out not to be designed but to emerge.\n$N$ blocks compose into the body. Each has identical shape but independent weights. Mechanistic-interpretability work finds a rough division of labor — early layers handle token-level and syntactic features, middle layers build relational/semantic structure, late layers assemble task- and output-specific features — but these roles are emergent, never hand-assigned. The output $H^{(N)}\\in\\mathbb R^{n\\times d}$ is the fully contextualized representation passed to the output head.\nLearned / config / state Quantity Category $W_1,W_2,\\mathbf b_1,\\mathbf b_2$ (MLP, per block) [LEARNED] $\\gamma,\\beta$ (norm, per block) [LEARNED] $N$, $d_{ff}$, choice of $\\phi$, norm type \u0026amp; placement, $\\epsilon$ [CONFIG] Residual additions [CONFIG] (no params) $\\mu,\\sigma$, hidden activations $\\phi(\\cdot)$ [STATE] (recomputed; cached for backward) Conclusion A transformer block is two residual sub-layers — multi-head attention and a position-wise MLP — each preceded by normalization (pre-norm). The MLP ($W_2\\phi(W_1\\mathbf x+\\mathbf b_1)+\\mathbf b_2$, width $d_{ff}=4d$) supplies the model\u0026rsquo;s nonlinear capacity (without $\\phi$ the whole network collapses to an affine map) and holds $\\approx\\tfrac23$ of parameters, giving $P\\approx12Nd^2+2Vd$. Residual connections make the layer Jacobian $I+\\partial\\mathcal F$, whose identity path carries gradients through arbitrarily deep stacks. LayerNorm/RMSNorm standardize each token\u0026rsquo;s features with a learned gain (and shift, for LayerNorm); their backward pass projects out mean/scale directions. Pre-norm placement is what makes deep training stable. Learned: MLP and norm weights. Config: depth, width, nonlinearity, norm type/placement. State: normalization statistics and hidden activations.\nThe network is now assembled from the bottom up — tokens became vectors, vectors learned to communicate, and the block stacked that communication into a deep body. What comes out the top, $H^{(N)}$, is a contextualized representation, not yet a prediction. Next we reach the top of the network and turn $H^{(N)}$ into a probability distribution.\n← Lesson 3 — Self-Attention · Course index · Next: Lesson 5 — Output Head \u0026amp; Objective → Appendices: A — Acronyms · B — Formulas · C — Attention Variants The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/tutorials/llm-lessons/lesson-04-the-transformer-block/","summary":"\u003ch1 id=\"lesson-4--the-transformer-block\"\u003eLesson 4 — The Transformer Block\u003c/h1\u003e\n\u003cp\u003e\u003cem\u003eTechnical course on LLM mechanics — Lesson 4 of 10.\u003c/em\u003e\u003c/p\u003e\n\u003chr\u003e\n\u003ch2 id=\"recap-of-lesson-3\"\u003eRecap of Lesson 3\u003c/h2\u003e\n\u003cp\u003eSelf-attention computes $A V$ with $A=\\mathrm{softmax}(QK^\\top/\\sqrt{d_k}+M)$, a row-stochastic matrix of learned asymmetric similarities; the $\\sqrt{d_k}$ divisor keeps softmax off its zero-gradient saturation (Jacobian $\\mathrm{diag}(\\mathbf a)-\\mathbf a\\mathbf a^\\top$); the causal mask enforces autoregression. Multi-head attention uses $h$ subspaces ($d_k=d/h$), costs $\\approx4d^2$ params/layer and $O(n^2d)$ compute. The attention matrix is \u003cstrong\u003e[STATE]\u003c/strong\u003e. We now assemble the full block and analyze depth, normalization, and the parameter budget.\u003c/p\u003e","title":"The Transformer Block"},{"content":"Lesson 4 — The Variational Lower Bound (ELBO) Recap of Lesson 3. Because forward steps are tiny, the reverse step is Gaussian; we model it as $p_\\theta(x_{t-1}\\mid x_t)=\\mathcal N(x_{t-1};\\mu_\\theta(x_t,t),\\Sigma_\\theta)$. The true reverse $q(x_{t-1}\\mid x_t)$ is intractable, but conditioning on $x_0$ yields a tractable Gaussian posterior $q(x_{t-1}\\mid x_t,x_0)=\\mathcal N(\\tilde\\mu_t,\\tilde\\beta_t I)$ with closed-form $\\tilde\\mu_t$ and $\\tilde\\beta_t$. Rewriting $\\tilde\\mu_t$ in terms of the noise $\\epsilon$ showed that predicting $\\epsilon$ recovers the ideal denoiser. This lesson explains why matching that posterior is the principled objective.\nThere is a number we badly want and cannot have: the likelihood the model assigns to real data. It hides behind an integral over every possible trajectory of noise, and that integral will not be computed in this lifetime. The move that rescues the whole enterprise is older than diffusion and quietly audacious — if you cannot measure the thing itself, measure something you can, and prove it never overshoots. This lesson builds that provable stand-in, the ELBO, and then patiently takes it apart until what is left is something a network can actually be trained on.\nOverview We have a target (the tractable posterior) and a model (the Gaussian reverse step), but not yet the loss that ties them together nor a guarantee that minimizing it improves $p_\\theta(x_0)$. The answer is the variational lower bound (ELBO), the latent-variable-model machinery behind the VAE (Kingma \u0026amp; Welling 2014), applied to a $T$-step chain. The marginal likelihood $p_\\theta(x_0)=\\int p_\\theta(x_{0:T})\\,dx_{1:T}$ is intractable, so we optimize a tractable bound; pushing the bound up raises the likelihood it supports.\nSetting up the bound We minimize $-\\log p_\\theta(x_0)$. Two routes give the same bound.\nRoute 1 — Jensen. Introduce the fixed forward process $q(x_{1:T}\\mid x_0)$ as the variational distribution. With $\\log$ concave,\n$$ -\\log p_\\theta(x_0) = -\\log \\int q(x_{1:T}\\mid x_0)\\,\\frac{p_\\theta(x_{0:T})}{q(x_{1:T}\\mid x_0)}\\,dx_{1:T} \\le \\mathbb{E}_{q}\\!\\left[-\\log\\frac{p_\\theta(x_{0:T})}{q(x_{1:T}\\mid x_0)}\\right] =: L. $$Route 2 — exact KL-gap identity. Write $p_\\theta(x_{0:T})=p_\\theta(x_{1:T}\\mid x_0)\\,p_\\theta(x_0)$ inside $L$:\n$$ L = \\mathbb{E}_q\\!\\left[-\\log\\frac{p_\\theta(x_{1:T}\\mid x_0)\\,p_\\theta(x_0)}{q(x_{1:T}\\mid x_0)}\\right] = -\\log p_\\theta(x_0) + \\mathbb{E}_q\\!\\left[\\log\\frac{q(x_{1:T}\\mid x_0)}{p_\\theta(x_{1:T}\\mid x_0)}\\right], $$where $-\\log p_\\theta(x_0)$ leaves the expectation as a constant w.r.t. $q$. Hence the exact gap is\n$$ \\boxed{\\ L - \\big(-\\log p_\\theta(x_0)\\big) = D_{\\mathrm{KL}}\\!\\big(q(x_{1:T}\\mid x_0)\\,\\|\\,p_\\theta(x_{1:T}\\mid x_0)\\big)\\ \\ge 0.\\ } $$KL non-negativity (Gibbs\u0026rsquo; inequality) re-derives Route 1 and makes the slack explicit: $L$ is tight iff the forward posterior equals the model\u0026rsquo;s reverse posterior. Thus $L$ upper-bounds the negative log-likelihood; $-L$ is the ELBO. Minimizing $L$ is the objective.\nFigure 1. A stand-in that provably never overshoots.\nDecomposing the bound into per-step KL terms A single intractable bound is progress, but not yet something to optimize; we now break it into pieces small enough to recognize one by one.\nExpand $L$ with the Markov factorizations of Lessons 2–3:\n$$ L = \\mathbb{E}_q\\!\\left[-\\log\\frac{p(x_T)\\prod_{t\\ge1}p_\\theta(x_{t-1}\\mid x_t)}{\\prod_{t\\ge1}q(x_t\\mid x_{t-1})}\\right] = \\mathbb{E}_q\\!\\left[-\\log p(x_T) - \\sum_{t\\ge1}\\log\\frac{p_\\theta(x_{t-1}\\mid x_t)}{q(x_t\\mid x_{t-1})}\\right]. $$Split off $t=1$ and, for $t\\ge2$, rewrite each forward factor with Bayes (using the Markov property $q(x_t\\mid x_{t-1})=q(x_t\\mid x_{t-1},x_0)$):\n$$ q(x_t\\mid x_{t-1}) = \\frac{q(x_{t-1}\\mid x_t, x_0)\\,q(x_t\\mid x_0)}{q(x_{t-1}\\mid x_0)}. $$Substituting, the $t\\ge2$ sum becomes\n$$ -\\sum_{t\\ge2}\\log\\frac{p_\\theta(x_{t-1}\\mid x_t)}{q(x_{t-1}\\mid x_t,x_0)} \\;-\\;\\sum_{t\\ge2}\\log\\frac{q(x_{t-1}\\mid x_0)}{q(x_t\\mid x_0)}. $$The telescoping ratio. The second sum telescopes. Writing the summand as $\\log q(x_{t-1}\\mid x_0)-\\log q(x_t\\mid x_0)$ for $t=2,\\dots,T$:\n$$ \\sum_{t=2}^{T}\\big[\\log q(x_{t-1}\\mid x_0)-\\log q(x_t\\mid x_0)\\big] = \\log q(x_1\\mid x_0) - \\log q(x_T\\mid x_0), $$because every interior term $\\log q(x_t\\mid x_0)$ (for $2\\le t\\le T-1$) appears once with each sign and cancels. The surviving $-\\big(-\\log q(x_1\\mid x_0)\\big)$ combines with the $t=1$ factor $-\\log\\frac{p_\\theta(x_0\\mid x_1)}{q(x_1\\mid x_0)}=-\\log p_\\theta(x_0\\mid x_1)+\\log q(x_1\\mid x_0)$: the two $\\log q(x_1\\mid x_0)$ pieces cancel, leaving only $-\\log p_\\theta(x_0\\mid x_1)$. The surviving $-\\log q(x_T\\mid x_0)$ pairs with $-\\log p(x_T)$ to form a KL. Taking $\\mathbb E_q$ and recognizing $\\mathbb E_{q}[-\\log\\frac{p_\\theta}{q}]=D_{\\mathrm{KL}}(q\\|p_\\theta)$ termwise:\n$$ \\boxed{, L = \\underbrace{D_{\\mathrm{KL}}!\\big(q(x_T\\mid x_0),|,p(x_T)\\big)}_{L_T}\n\\sum_{t=2}^{T}\\underbrace{D_{\\mathrm{KL}}!\\big(q(x_{t-1}\\mid x_t,x_0),|,p_\\theta(x_{t-1}\\mid x_t)\\big)}{L{t-1}} \\underbrace{-,\\log p_\\theta(x_0\\mid x_1)}_{L_0} ,} $$ A concrete walk-through: telescoping at $T=3$ The general argument above asks you to hold several identities in your head at once. It is worth doing once with all the steps written out on the smallest non-trivial chain, $T=3$. Nothing is hidden here — every term is tracked until it either survives or cancels.\nStep 0 — write $L$ explicitly. With $T=3$ the sum $\\sum_{t\\ge1}$ runs over $t=1,2,3$, so the expression inside $\\mathbb E_q$ (call it $A$, so $L=\\mathbb E_q[A]$) is\n$$ A = -\\log p(x_3) \\;\\underbrace{-\\log\\frac{p_\\theta(x_0\\mid x_1)}{q(x_1\\mid x_0)}}_{t=1} \\;\\underbrace{-\\log\\frac{p_\\theta(x_1\\mid x_2)}{q(x_2\\mid x_1)}}_{t=2} \\;\\underbrace{-\\log\\frac{p_\\theta(x_2\\mid x_3)}{q(x_3\\mid x_2)}}_{t=3}. $$Step 1 — leave $t=1$ alone; rewrite the forward factors for $t=2,3$ with Bayes. Using $q(x_t\\mid x_{t-1})=\\dfrac{q(x_{t-1}\\mid x_t,x_0)\\,q(x_t\\mid x_0)}{q(x_{t-1}\\mid x_0)}$:\n$$ \\log q(x_2\\mid x_1)=\\log q(x_1\\mid x_2,x_0)+\\log q(x_2\\mid x_0)-\\log q(x_1\\mid x_0), $$ $$ \\log q(x_3\\mid x_2)=\\log q(x_2\\mid x_3,x_0)+\\log q(x_3\\mid x_0)-\\log q(x_2\\mid x_0). $$Step 2 — substitute and group by type. Expanding $A$ and colour-sorting the pieces:\n$$ A = \\underbrace{-\\log p_\\theta(x_0\\mid x_1)}_{\\to\\,L_0} \\;\\underbrace{-\\log\\frac{p_\\theta(x_1\\mid x_2)}{q(x_1\\mid x_2,x_0)}}_{\\to\\,L_1} \\;\\underbrace{-\\log\\frac{p_\\theta(x_2\\mid x_3)}{q(x_2\\mid x_3,x_0)}}_{\\to\\,L_2} \\;-\\log p(x_3) \\;+\\;\\Big[\\,\\underbrace{\\log q(x_1\\mid x_0)}_{\\text{from }t=1}+\\log q(x_2\\mid x_0)-\\log q(x_1\\mid x_0)+\\log q(x_3\\mid x_0)-\\log q(x_2\\mid x_0)\\Big]. $$Step 3 — watch the marginal-log terms cancel. Inside the bracket, line up the $\\log q(x_t\\mid x_0)$ pieces:\nterm sign(s) it appears with fate $\\log q(x_1\\mid x_0)$ $+$ (from the $t=1$ factor) and $-$ (from the $t=2$ Bayes step) cancels $\\log q(x_2\\mid x_0)$ $+$ (from $t=2$) and $-$ (from $t=3$) cancels (an interior term) $\\log q(x_3\\mid x_0)$ $+$ (from $t=3$) only survives So the entire bracket collapses to a single survivor, $\\log q(x_3\\mid x_0)$.\nStep 4 — read off the four groups. Pairing the survivor with $-\\log p(x_3)$ and taking $\\mathbb E_q$ (so each $\\mathbb E_q[-\\log\\frac{p}{q}]$ becomes a $D_{\\mathrm{KL}}(q\\,\\|\\,p)$):\n$$ L = \\underbrace{D_{\\mathrm{KL}}\\!\\big(q(x_3\\mid x_0)\\,\\|\\,p(x_3)\\big)}_{L_T,\\ \\ T=3} \\;+\\;\\underbrace{D_{\\mathrm{KL}}\\!\\big(q(x_2\\mid x_3,x_0)\\,\\|\\,p_\\theta(x_2\\mid x_3)\\big)}_{L_2} \\;+\\;\\underbrace{D_{\\mathrm{KL}}\\!\\big(q(x_1\\mid x_2,x_0)\\,\\|\\,p_\\theta(x_1\\mid x_2)\\big)}_{L_1} \\;\\underbrace{-\\,\\log p_\\theta(x_0\\mid x_1)}_{L_0}. $$This is exactly the boxed formula with $T=3$: one prior term $L_T$, the denoising sum $\\sum_{t=2}^{3}L_{t-1}=L_2+L_1$, and the reconstruction term $L_0$. The two cancellations in Step 3 are the whole content of \u0026ldquo;the telescoping ratio\u0026rdquo;; for general $T$ the same bookkeeping leaves every interior $\\log q(x_t\\mid x_0)$ paired off, with only the endpoints $\\log q(x_1\\mid x_0)$ (absorbed into $L_0$) and $\\log q(x_T\\mid x_0)$ (forming $L_T$) untouched.\nEach piece has a clear role:\n$L_T$ — prior matching. $D_{\\mathrm{KL}}(q(x_T\\mid x_0)\\,\\|\\,\\mathcal N(0,I))$ has no learnable parameters (forward process fixed), so it is a constant in $\\theta$, ignorable during optimization, and is small by design ($\\bar\\alpha_T\\approx0$). $L_{t-1}$ — denoising matching (the heart). Each compares the model reverse step to the tractable posterior of Lesson 3; the $T-1$ terms carry essentially all learning signal. $L_0$ — reconstruction. Decoding $x_1\\to x_0$ to discrete data, handled by a separate discretized-likelihood model (Ho et al. 2020). Figure 2. What the telescoping leaves behind.\nEvaluating the denoising terms: Gaussian KL Each surviving piece is a divergence between two Gaussians — and that is a piece of real luck, because there are few quantities in all of statistics we can write down more explicitly.\nThe decomposition pays off because both arguments of each $L_{t-1}$ are Gaussians. For two $d$-dimensional Gaussians the KL is closed-form. General diagonal case $\\mathcal N(\\mu_1,\\Sigma_1)\\,\\|\\,\\mathcal N(\\mu_2,\\Sigma_2)$:\n$$ D_{\\mathrm{KL}} = \\frac{1}{2}\\!\\left[\\operatorname{tr}(\\Sigma_2^{-1}\\Sigma_1) + (\\mu_2-\\mu_1)^\\top\\Sigma_2^{-1}(\\mu_2-\\mu_1) - d + \\log\\frac{\\det\\Sigma_2}{\\det\\Sigma_1}\\right]. $$Equal covariance $\\Sigma_1=\\Sigma_2=\\Sigma$: the trace term equals $d$, the log-det term vanishes, and only the Mahalanobis mean gap survives:\n$$ D_{\\mathrm{KL}}\\!\\big(\\mathcal N(\\tilde\\mu,\\Sigma)\\,\\|\\,\\mathcal N(\\mu_\\theta,\\Sigma)\\big) = \\frac{1}{2}\\,(\\tilde\\mu - \\mu_\\theta)^\\top \\Sigma^{-1} (\\tilde\\mu - \\mu_\\theta). $$Isotropic specialization $\\Sigma=\\sigma_t^2 I$, so $\\Sigma^{-1}=\\sigma_t^{-2}I$:\n$$ \\boxed{\\ L_{t-1} = \\frac{1}{2\\sigma_t^2}\\,\\big\\|\\tilde\\mu_t(x_t,x_0) - \\mu_\\theta(x_t,t)\\big\\|^2 + C.\\ } $$DDPM fixes the reverse covariance to a schedule constant, $\\Sigma_\\theta=\\sigma_t^2 I$ with $\\sigma_t^2=\\beta_t$ or $\\sigma_t^2=\\tilde\\beta_t$ (comparable in practice), which is exactly the equal-covariance condition that drops every term but the mean gap. The grand variational objective has reduced to mean-matching regression weighted by $1/2\\sigma_t^2$.\nRigor remark — learned variance and hybrid loss. Fixing $\\sigma_t^2$ is a modeling choice. Improved DDPM (Nichol \u0026amp; Dhariwal 2021) learns the reverse variance as a per-coordinate interpolation in log-space, $\\Sigma_\\theta=\\exp\\!\\big(v\\log\\beta_t+(1-v)\\log\\tilde\\beta_t\\big)$, with $v=v_\\theta(x_t,t)\\in[0,1]$ keeping the variance bracketed by the two natural bounds $\\tilde\\beta_t\\le\\Sigma_\\theta\\le\\beta_t$. With learned $\\Sigma_\\theta$ the full Gaussian KL (trace and log-det terms) no longer cancels, so $L_{t-1}$ must be evaluated in full. They optimize a hybrid loss $L_{\\text{hybrid}}=L_{\\text{simple}}+\\lambda L_{\\text{vlb}}$ (typically $\\lambda=10^{-3}$), with a stop-gradient on $\\mu_\\theta$ inside $L_{\\text{vlb}}$ so the simple loss trains the mean and the VLB term trains only the variance. This improves log-likelihood (NLL) markedly with negligible sample-quality cost. The continuous-time VDM treatment (Kingma et al. 2021) recasts the bound as an SNR-weighted integral and shows the weighting, not the parameterization, is what matters.\nNumerical note. Evaluate the full Gaussian KL in log-variance space and use closed forms (tr, log det as sums of diagonal logs) rather than forming $\\Sigma^{-1}$. At $t=1$, $\\tilde\\beta_1=0$, so $L_0$ is the discretized decoder, not a KL; do not feed a degenerate Gaussian into the KL formula.\nFigure 3. Why fixing the reverse variance turns a KL into a regression.\nExercises 1. The exact KL gap. Starting from $L=\\mathbb E_q[-\\log\\frac{p_\\theta(x_{0:T})}{q(x_{1:T}\\mid x_0)}]$, use $p_\\theta(x_{0:T})=p_\\theta(x_{1:T}\\mid x_0)\\,p_\\theta(x_0)$ to show $L=-\\log p_\\theta(x_0)+D_{\\mathrm{KL}}(q(x_{1:T}\\mid x_0)\\,\\|\\,p_\\theta(x_{1:T}\\mid x_0))$, and hence $L\\ge-\\log p_\\theta(x_0)$.\nSubstituting the factorization,\n$$ L=\\mathbb E_q\\!\\left[-\\log\\frac{p_\\theta(x_{1:T}\\mid x_0)\\,p_\\theta(x_0)}{q(x_{1:T}\\mid x_0)}\\right] =-\\log p_\\theta(x_0)+\\mathbb E_q\\!\\left[\\log\\frac{q(x_{1:T}\\mid x_0)}{p_\\theta(x_{1:T}\\mid x_0)}\\right], $$because $-\\log p_\\theta(x_0)$ has no $x_{1:T}$ dependence and leaves the expectation. The remaining expectation is exactly $D_{\\mathrm{KL}}(q(x_{1:T}\\mid x_0)\\,\\|\\,p_\\theta(x_{1:T}\\mid x_0))\\ge0$ by Gibbs\u0026rsquo; inequality, so $L\\ge-\\log p_\\theta(x_0)$.\n2. Telescope at $T=2$. Redo the worked walk-through for the minimal chain $T=2$ and show $L=D_{\\mathrm{KL}}(q(x_2\\mid x_0)\\,\\|\\,p(x_2))+D_{\\mathrm{KL}}(q(x_1\\mid x_2,x_0)\\,\\|\\,p_\\theta(x_1\\mid x_2))-\\log p_\\theta(x_0\\mid x_1)$. Which cancellation from the $T=3$ case is now absent?\nFor $T=2$ the sum runs over $t=1,2$. The $t=1$ factor keeps $+\\log q(x_1\\mid x_0)$; the $t=2$ Bayes step gives $+\\log q(x_1\\mid x_2,x_0)+\\log q(x_2\\mid x_0)-\\log q(x_1\\mid x_0)$. The two $\\log q(x_1\\mid x_0)$ pieces cancel; $\\log q(x_2\\mid x_0)$ pairs with $-\\log p(x_2)$ to form $L_T$; $-\\log p_\\theta(x_1\\mid x_2)+\\log q(x_1\\mid x_2,x_0)$ is $L_1$; and $-\\log p_\\theta(x_0\\mid x_1)$ is $L_0$. There is no interior cancellation (no $\\log q(x_2\\mid x_0)$-type term appearing with both signs) — $T=2$ is the smallest chain, so only the endpoint cancellation occurs.\n3. Equal-covariance KL collapse. Starting from the general diagonal-Gaussian KL, set $\\Sigma_1=\\Sigma_2=\\Sigma$ and show it reduces to $\\tfrac12(\\mu_2-\\mu_1)^\\top\\Sigma^{-1}(\\mu_2-\\mu_1)$.\nWith $\\Sigma_1=\\Sigma_2=\\Sigma$: the trace term $\\operatorname{tr}(\\Sigma^{-1}\\Sigma)=\\operatorname{tr}(I)=d$ cancels the $-d$; the log-det term $\\log\\frac{\\det\\Sigma}{\\det\\Sigma}=0$. Only the Mahalanobis mean-gap term $\\tfrac12(\\mu_2-\\mu_1)^\\top\\Sigma^{-1}(\\mu_2-\\mu_1)$ survives — which is why fixing the reverse covariance to a schedule constant turns each $L_{t-1}$ into pure mean-matching.\n4. Why $L_T$ is free. Explain why $L_T$ carries no learnable parameters, and why it is small by design.\n$L_T=D_{\\mathrm{KL}}(q(x_T\\mid x_0)\\,\\|\\,p(x_T))$ compares the fixed forward marginal $q(x_T\\mid x_0)$ (no $\\theta$) with the fixed prior $p(x_T)=\\mathcal N(0,I)$ (no $\\theta$), so it is a constant in $\\theta$ — ignorable for optimization. It is small because the schedule drives $\\bar\\alpha_T\\approx0$, making $q(x_T\\mid x_0)\\approx\\mathcal N(0,I)$, i.e. nearly the prior, so the KL $\\to0$.\nConclusion Maximizing the data likelihood is intractable, so we maximize a variational lower bound; the slack is exactly $D_{\\mathrm{KL}}(q(x_{1:T}\\mid x_0)\\,\\|\\,p_\\theta(x_{1:T}\\mid x_0))$. Rewriting the forward factors with the $x_0$-conditioned posterior and telescoping the $q(x_t\\mid x_0)/q(x_{t-1}\\mid x_0)$ ratios decomposes $L$ into a constant prior term $L_T$, a reconstruction term $L_0$, and the heart — a sum of closed-form Gaussian KL denoising terms $L_{t-1}$. With fixed isotropic reverse variance these collapse to $\\frac{1}{2\\sigma_t^2}\\|\\tilde\\mu_t-\\mu_\\theta\\|^2$: mean-matching regression. Optional learned variance (Improved DDPM) trades the cancellation for better likelihood via a hybrid loss. In Lesson 5 we substitute the noise parameterization, watch the means cancel, and obtain the one-line DDPM training loss.\nWe set out to measure something unmeasurable and ended up holding a tidy sum of Gaussian KLs — a constant we can ignore, a reconstruction term, and a heart made of mean-matching. Lesson 5 watches that heart collapse, almost magically, into a single line of regression.\nReferences J. Ho, A. Jain, P. Abbeel. Denoising Diffusion Probabilistic Models. NeurIPS 2020. J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, S. Ganguli. Deep Unsupervised Learning using Nonequilibrium Thermodynamics. ICML 2015. D. P. Kingma, M. Welling. Auto-Encoding Variational Bayes. ICLR 2014 (ELBO / VAE). A. Nichol, P. Dhariwal. Improved Denoising Diffusion Probabilistic Models. ICML 2021 (learned variance, hybrid loss). D. P. Kingma, T. Salimans, B. Poole, J. Ho. Variational Diffusion Models. NeurIPS 2021 (continuous-time / SNR-weighted bound). ← Previous: Lesson 3 — The Reverse Process · Course Index · Next: Lesson 5 — The Simplified Objective → · Appendix A: Acronyms · Appendix B: Formulas The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/tutorials/diffusion-lessons/lesson-04-variational-lower-bound/","summary":"\u003ch1 id=\"lesson-4--the-variational-lower-bound-elbo\"\u003eLesson 4 — The Variational Lower Bound (ELBO)\u003c/h1\u003e\n\u003cblockquote\u003e\n\u003cp\u003e\u003cstrong\u003eRecap of Lesson 3.\u003c/strong\u003e Because forward steps are tiny, the reverse step is Gaussian; we model it as $p_\\theta(x_{t-1}\\mid x_t)=\\mathcal N(x_{t-1};\\mu_\\theta(x_t,t),\\Sigma_\\theta)$. The true reverse $q(x_{t-1}\\mid x_t)$ is intractable, but conditioning on $x_0$ yields a tractable Gaussian posterior $q(x_{t-1}\\mid x_t,x_0)=\\mathcal N(\\tilde\\mu_t,\\tilde\\beta_t I)$ with closed-form $\\tilde\\mu_t$ and $\\tilde\\beta_t$. Rewriting $\\tilde\\mu_t$ in terms of the noise $\\epsilon$ showed that predicting $\\epsilon$ recovers the ideal denoiser. This lesson explains \u003cem\u003ewhy\u003c/em\u003e matching that posterior is the principled objective.\u003c/p\u003e","title":"The Variational Lower Bound"},{"content":"Lesson 5 — Load Balancing \u0026amp; Auxiliary Losses Technical course on MoE LLM mechanics — Lesson 5 of 10.\nRecap of Lesson 4 The MoE forward pass computes only the $k$ selected experts and combines them as $\\mathbf y=\\sum_{i\\in\\mathcal T_k}g_iE_i(\\mathbf x)$ — sparse in compute, dense in memory. Backward, only selected experts and only selected router logits receive gradient; the router gets a relative reweighting of its winners and never a signal to choose differently. That partial, self-reinforcing coverage means a popular expert gets more training, gets better, and gets chosen more — a positive feedback loop toward collapse. This lesson names that failure, measures it, and adds the explicit counter-pressure that is the defining peculiarity of MoE training.\nEverything so far would train — badly. Left to the cross-entropy loss alone, the router has no incentive to use all the experts and every incentive to concentrate. This lesson is about the extra term in the loss that exists for one reason: to make a sparse model use its sparsity well.\n1. Expert collapse: the failure mode Suppose, by chance of initialization, a few experts are slightly better early on. The router sends them more tokens; those tokens give them more gradient (Lesson 4); they improve faster; the router sends them even more. Meanwhile the neglected experts receive few or no tokens, get little or no gradient, never improve, and fall further out of favor. The endpoint is expert collapse (also \u0026ldquo;dead experts\u0026rdquo;): a handful of experts do almost all the work and the rest are dead weight — parameters that occupy memory but are never meaningfully trained or used.\nCollapse is catastrophic for an MoE specifically because the entire value proposition was capacity. If $E=64$ but only $4$ experts are ever used, you have paid the memory of a 64-expert model for the capacity of a 4-expert one. Worse, at training time imbalance also wrecks efficiency: experts are spread across devices (Lesson 7) with a fixed compute budget each (Lesson 6), so a few overloaded experts become a throughput bottleneck while idle devices wait. Balance is both a quality issue and a systems issue.\nThe cross-entropy objective will not fix this on its own — a collapsed router can still drive the loss down using its few good experts, so there is no gradient pressure toward balance from the task loss. Balance must be added as a separate objective.\nFigure 1. The feedback loop that ends in dead experts.\n2. What we want to encourage For a batch of $T$ tokens routed over $E$ experts, define two per-expert quantities ([STATE], computed each batch):\nthe fraction of tokens dispatched to expert $i$, $$ f_i=\\frac{1}{T}\\sum_{t=1}^{T}\\mathbb{1}\\!\\left[i\\in\\mathcal T_k(\\mathbf x_t)\\right], $$ the average router probability mass on expert $i$, $$ P_i=\\frac{1}{T}\\sum_{t=1}^{T} P_i(\\mathbf x_t),\\qquad P_i(\\mathbf x_t)=\\mathrm{softmax}(W_g\\mathbf x_t)_i. $$ $f_i$ is the hard load (a count of assignments); $P_i$ is the soft load (the probability the router would assign). Perfect balance is $f_i=P_i=1/E$ for all $i$. We want a penalty that is minimized at uniformity and that, crucially, is differentiable in $W_g$ so it can actually steer the router — and that is why we cannot penalize $f_i$ alone ($f_i$ comes from the non-differentiable top-$k$, Lesson 3).\nFigure 2. The load profile we have, and the one we want.\n3. The Switch Transformer auxiliary loss The standard, simple choice (Switch Transformer) couples the non-differentiable load $f_i$ with the differentiable mass $P_i$ in a single product, summed over experts:\n$$ \\boxed{\\ \\mathcal L_{\\text{aux}}=\\alpha\\,E\\sum_{i=1}^{E} f_i\\,P_i\\ } $$with a small [CONFIG] weight $\\alpha$ (Switch uses $\\alpha=10^{-2}$). The factor $E$ normalizes so the loss is $1$ at perfect balance. The total training objective is\n$$ \\mathcal L=\\mathcal L_{\\text{CE}}+\\mathcal L_{\\text{aux}}, $$the ordinary cross-entropy plus this penalty (summed over all MoE layers).\nWhy this form works. $\\mathcal L_{\\text{aux}}$ is a dot product $E\\,\\mathbf f^\\top\\mathbf P$ between the load vector and the mass vector, both of which sum to a constant ($\\sum_i f_i=k$, $\\sum_i P_i=1$). Such a dot product, under a sum constraint, is minimized when the mass is spread away from the heavily-loaded experts — pushing $\\mathbf P$ toward uniform. The gradient flows through the differentiable factor $P_i$ (we treat $f_i$ as a constant multiplier for the backward pass, since it has no usable gradient): an expert with high load $f_i$ gets its router mass $P_i$ pushed down, which over many batches makes the router stop over-selecting it. It is a beautifully cheap trick — it makes a non-differentiable quantity (the load) steer a differentiable one (the mass) by multiplying them together.\nMinimizing $\\mathbf f^\\top\\mathbf P$ subject to $\\sum_i f_i=k$ and $\\sum_i P_i=1$ with both non-negative is driven to its lower bound exactly when load and mass are both uniform, $f_i=k/E$ and $P_i=1/E$ — the balanced state we wanted.\n4. The router z-loss: keeping logits small A second, complementary auxiliary term addresses stability rather than balance. Router logits can grow large during training, which makes the softmax sharp, the routing brittle, and the exponentials numerically dangerous in low precision (Lesson 8). The router z-loss (ST-MoE) penalizes the log-partition function of the router softmax:\n$$ \\mathcal L_{z}=\\beta\\,\\frac{1}{T}\\sum_{t=1}^{T}\\left(\\log\\sum_{i=1}^{E} e^{h_i(\\mathbf x_t)}\\right)^{2}, $$with a small [CONFIG] weight $\\beta$. The quantity $\\log\\sum_i e^{h_i}$ is the softmax normalizer (the \u0026ldquo;$z$\u0026rdquo;); squaring and penalizing it pulls the logits toward a moderate magnitude, keeping the router in a well-conditioned, numerically safe regime without dictating which expert wins. The full MoE objective is then\n$$ \\mathcal L=\\mathcal L_{\\text{CE}}+\\alpha E\\sum_i f_iP_i+\\beta\\,\\overline{(\\log\\textstyle\\sum_i e^{h_i})^2}. $$ 5. Aux-loss-free balancing (the modern alternative) Auxiliary losses have a cost: they are a second objective fighting the task loss, and too large an $\\alpha$ trades away model quality for balance. Recent models (notably DeepSeek-V3) replace the loss with a bias-adjusted routing scheme that needs no extra gradient term. Each expert gets a scalar bias $b_i$ added to its logit for selection purposes only:\n$$ \\mathcal T_k(\\mathbf x)=\\arg\\text{top-}k_i\\,\\big(h_i(\\mathbf x)+b_i\\big), $$while the gate weights $g_i$ still use the unbiased logits. The biases are not trained by gradient descent; they are nudged by a simple control rule between steps — decrease $b_i$ for over-loaded experts, increase it for under-loaded ones — so over time selection equalizes:\n$$ b_i\\leftarrow b_i-\\gamma\\,\\mathrm{sign}\\big(f_i-\\tfrac{k}{E}\\big), $$with a small update rate $\\gamma$ ([CONFIG]). Because the bias enters only the discrete selection and never the differentiable combine, it steers load without adding a competing gradient to the task loss — balance is achieved while leaving the cross-entropy objective pure. This is increasingly the preferred approach at large scale.\nContrast with the dense model. A dense model\u0026rsquo;s loss is just the task loss $\\mathcal L_{\\text{CE}}$ — there is nothing to balance, because there is no choice to make; every parameter is used equally by construction. An MoE cannot be trained on the task loss alone: it needs an explicit balancing mechanism (auxiliary loss, z-loss, or bias control) bolted onto the objective or the routing rule. This extra machinery — and the hyperparameters $\\alpha,\\beta,\\gamma$ that tune it — has no analogue whatsoever in dense training. It is the single biggest way MoE training differs from dense training.\nFigure 3. Three ways to make a sparse model use its sparsity.\n6. Learned / config / state Quantity Category Router $W_g$, expert weights [LEARNED] Aux-loss weight $\\alpha$, z-loss weight $\\beta$, bias update rate $\\gamma$, balancing scheme [CONFIG] Selection biases $b_i$ (control-updated, not gradient-trained) [STATE]-like control variable (persisted across steps) Per-batch load $f_i$, mass $P_i$, logits $h_i$ [STATE] Conclusion Sparse routing has a built-in failure mode — expert collapse, where a positive feedback loop funnels tokens to a few experts and starves the rest, destroying both the capacity you paid for and training throughput. Because the task cross-entropy gives no pressure toward balance, balance must be added. The Switch auxiliary loss $\\mathcal L_{\\text{aux}}=\\alpha E\\sum_i f_iP_i$ multiplies the non-differentiable load $f_i$ by the differentiable mass $P_i$ so the router is pushed toward uniform usage; the router z-loss $\\beta\\,\\overline{(\\log\\sum_i e^{h_i})^2}$ keeps logits small and routing numerically stable; and aux-loss-free bias balancing ($\\mathcal T_k$ on $h_i+b_i$, biases nudged by a control rule) achieves balance without a competing gradient. This balancing apparatus — and its hyperparameters $\\alpha,\\beta,\\gamma$ — is the defining difference between MoE and dense training; nothing like it exists in a dense model.\nBalancing the router\u0026rsquo;s intent is half the story. Even a perfectly intentioned router must fit its tokens into experts that live on hardware with finite, fixed-size buffers — which forces the notion of capacity and the possibility of dropping tokens. That is the next lesson.\n← Lesson 4 — Sparse Top-k Computation \u0026amp; the Combine · Course index · Next: Lesson 6 — Capacity \u0026amp; Token Dropping → Appendices: A — MoE Family · B — Dense vs. MoE · C — Acronyms The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/tutorials/moe-lessons/lesson-05-load-balancing-and-auxiliary-losses/","summary":"\u003ch1 id=\"lesson-5--load-balancing--auxiliary-losses\"\u003eLesson 5 — Load Balancing \u0026amp; Auxiliary Losses\u003c/h1\u003e\n\u003cp\u003e\u003cem\u003eTechnical course on MoE LLM mechanics — Lesson 5 of 10.\u003c/em\u003e\u003c/p\u003e\n\u003chr\u003e\n\u003ch2 id=\"recap-of-lesson-4\"\u003eRecap of Lesson 4\u003c/h2\u003e\n\u003cp\u003eThe MoE forward pass computes only the $k$ selected experts and combines them as $\\mathbf y=\\sum_{i\\in\\mathcal T_k}g_iE_i(\\mathbf x)$ — sparse in compute, dense in memory. Backward, only selected experts and only selected router logits receive gradient; the router gets a \u003cem\u003erelative reweighting of its winners\u003c/em\u003e and never a signal to choose differently. That partial, self-reinforcing coverage means a popular expert gets more training, gets better, and gets chosen more — a positive feedback loop toward collapse. This lesson names that failure, measures it, and adds the explicit counter-pressure that is \u003cstrong\u003ethe defining peculiarity of MoE training.\u003c/strong\u003e\u003c/p\u003e","title":"Load Balancing and Auxiliary Losses"},{"content":"Lesson 5 — Masked Language Modeling Technical course on encoder LLM mechanics — Lesson 5 of 10.\nRecap of Lesson 4 Stacking $N$ bidirectional blocks turns the input $H^{(0)}$ into $H^{(N)}\\in\\mathbb R^{n\\times d}$, a contextual vector per token plus a [CLS] summary register — the encoder\u0026rsquo;s entire output, with no predictions attached. But why does the stack learn useful representations? A randomly initialized encoder produces garbage; the representations become meaningful only because pretraining pushes a learning signal through every weight. This lesson derives that signal. It is the single biggest departure from the decoder, and Lesson 3 already told us why one is needed: bidirectionality kills next-token prediction.\n1. The objective an encoder cannot use, and the one it can A decoder is pretrained on the next-token loss\n$$ \\mathcal L_{\\text{AR}}(\\theta)=-\\,\\mathbb E_{x\\sim\\mathcal D}\\sum_{t=1}^{n}\\log P_\\theta\\big(x_t\\mid x_{\\lt t}\\big), $$which works precisely because causal masking hides $x_{\\ge t}$ from the predictor of $x_t$. Lesson 3 showed this collapses for a bidirectional model: the predictor of $x_t$ has attended to $x_t$ itself, so the task is trivially solved by copying. We need an objective that (a) is hard even with full bidirectional context and (b) forces the model to build representations that understand each token from its surroundings.\nBERT\u0026rsquo;s answer is masked language modeling (MLM), also called the cloze task or a denoising autoencoder objective: hide some tokens, then reconstruct them from everything else. The model never sees the answer for a position it must predict, so bidirectional context is an asset (it can use both sides) rather than a leak.\nInvariant. A decoder predicts the next token from the past; an encoder predicts a hidden token from both sides. Same loss function (cross-entropy), opposite information structure.\n2. The MLM objective, formally Let $x=(x_1,\\dots,x_n)$ be an input and $\\mathcal M\\subset\\{1,\\dots,n\\}$ a randomly chosen set of positions to corrupt, with $|\\mathcal M|\\approx 0.15\\,n$ (the masking rate is a [CONFIG], classically 15%). Write $\\tilde x$ for the corrupted sequence (§3 says exactly how positions in $\\mathcal M$ are corrupted). Run the encoder on $\\tilde x$, take the top-layer vectors $H^{(N)}=g(\\tilde x;\\theta)$, and at each masked position $i\\in\\mathcal M$ predict a distribution over the vocabulary via the MLM head (Lesson 6):\n$$ \\hat{\\mathbf p}_i=\\mathrm{softmax}\\big(W_{\\text{MLM}}\\,\\phi(H^{(N)}_{i,:})+\\mathbf b\\big)\\in\\Delta^{V-1}. $$The loss is cross-entropy, summed only over masked positions:\n$$ \\boxed{\\ \\mathcal L_{\\text{MLM}}(\\theta)=-\\,\\mathbb E_{x\\sim\\mathcal D}\\ \\mathbb E_{\\mathcal M}\\ \\sum_{i\\in\\mathcal M}\\log P_\\theta\\big(x_i\\mid \\tilde x\\big).\\ } $$Compare the two boxed losses term by term. The conditioning set is $\\tilde x$ (the whole corrupted sequence, both sides) rather than $x_{\\lt t}$ (the past). The summation index is $i\\in\\mathcal M$ (only the ~15% masked positions) rather than $t\\in\\{1,\\dots,n\\}$ (every position). Both differences matter, and the second has a sharp efficiency consequence we treat in §5 and Lesson 8.\nflowchart LR A[\"the cat sat on the mat\"] --\u003e B[\"corrupt 15%:the [MASK] sat on the mat\"] B --\u003e C[\"bidirectional encoder\"] C --\u003e D[\"at masked position:predict P(· | rest) over V\"] D --\u003e E[\"loss = −log P('cat')(masked positions only)\"] Figure 1. The masked-reconstruction loop, on the lesson\u0026rsquo;s example.\n3. The 80/10/10 corruption recipe and the reason for it The obvious implementation — always replace a chosen token with [MASK] — has a subtle but serious flaw: the [MASK] token appears only during pretraining. At fine-tuning and inference time (Lesson 9) there is no [MASK]; the model sees real, intact text. If the model only ever learned to build good representations at [MASK] positions, it would be mismatched to the downstream regime where no position is masked. This is a train/test (pretrain/finetune) distribution mismatch.\nBERT mitigates it with a three-way corruption. For each position selected for the masked set (the 15%):\nwith probability 80%, replace it with [MASK]; with probability 10%, replace it with a random token from the vocabulary; with probability 10%, leave it unchanged. In all three cases the position is still scored (it is in $\\mathcal M$, so it contributes to the loss and the model must predict the original token). The logic:\nThe 80% [MASK] case is the core denoising task. The 10% random-token case forces the model not to blindly trust the surface token at a scored position — it might be wrong — so it must rely on context to decide. This counteracts the model becoming complacent. The 10% unchanged case means the model must produce a good representation/prediction even for a token that looks completely normal, which is exactly the inference-time situation (every token is real). This narrows the train/test gap. Because the model cannot tell, from the surface form alone, whether a scored token was masked, swapped, or kept, it is pushed to build a strong contextual representation at every position — which is what we want for downstream tasks.\n[CONFIG] The 15% rate and the 80/10/10 split are hyperparameters, not learned. They are classic BERT values; later work (Appendix A) revisits them — e.g. some studies mask up to 40% with large models, and ModernBERT tunes the rate.\nContrast with the decoder. A decoder has no train/test masking gap to worry about: it is trained and deployed on the same left-to-right token stream, scoring every position. The encoder\u0026rsquo;s 80/10/10 recipe exists entirely to paper over an artifact — the [MASK] token — that the decoder never introduces. ELECTRA (Appendix A) sidesteps the gap altogether with a different objective.\nFigure 2. The corruption recipe, position by position.\n4. Next-Sentence Prediction (NSP), and why it was dropped Original BERT added a second pretraining objective alongside MLM, aimed at sentence-pair tasks (entailment, QA). Given two segments A and B packed as [CLS] A [SEP] B [SEP], the model predicts a binary label from the [CLS] vector: is B the actual next sentence after A (positive), or a random sentence from the corpus (negative)?\n$$ \\mathcal L_{\\text{NSP}}=-\\big[y\\log\\sigma(\\mathbf w^\\top H^{(N)}_{0,:})+(1-y)\\log(1-\\sigma(\\cdot))\\big], $$and BERT minimized $\\mathcal L_{\\text{MLM}}+\\mathcal L_{\\text{NSP}}$. NSP is the original reason [CLS] and the segment embeddings (Lesson 2) earn their keep during pretraining.\nThe catch: later work (RoBERTa, ALBERT) found NSP does not help, and can hurt. The task is too easy — distinguishing a coherent continuation from a random sentence often reduces to topic detection, which MLM already learns — and removing it (training on longer contiguous spans instead) improved downstream results. ALBERT replaced it with Sentence-Order Prediction (SOP): both candidates are real adjacent sentences, and the model predicts whether they are in the correct order — a genuinely harder, more useful coherence signal. The modern consensus: MLM alone (with good data and long contexts) is the workhorse; NSP was a misstep.\nContrast with the decoder. The decoder never needed an explicit sentence-coherence objective — next-token prediction over long documents implicitly teaches discourse coherence for free. NSP was an attempt to inject, by hand, a signal that autoregressive training gets as a byproduct, and it turned out to be unnecessary.\nFigure 3. NSP, SOP, and the objective that survived.\n5. The efficiency price of masking only 15% One quantitative consequence deserves attention now (Lesson 8 returns to it). A decoder\u0026rsquo;s forward pass produces a training signal at every one of the $n$ positions — every token is a prediction target. An MLM encoder runs the same-cost forward pass but only the ~15% masked positions contribute to the loss; the other ~85% of positions are processed but produce no direct learning signal. So per forward-pass FLOP, MLM extracts roughly $1/0.15\\approx 6\\times$ less supervision than autoregressive training.\nThis is the central efficiency critique of MLM and the motivation for ELECTRA\u0026rsquo;s replaced-token detection, which scores all positions (Appendix A). It does not make MLM bad — bidirectional representations are worth a great deal for understanding tasks — but it explains why MLM pretraining is comparatively data-hungry and why later encoders work hard to raise the signal density.\nTakeaway. The \u0026ldquo;predict only masked positions\u0026rdquo; design is what makes the encoder objective valid under bidirectionality, but it is also what makes it sample-inefficient relative to the decoder\u0026rsquo;s \u0026ldquo;predict every position.\u0026rdquo; Bidirectional context vs. dense supervision is the core trade.\nConclusion Because bidirectional attention makes next-token prediction trivial (the answer is visible), an encoder is pretrained by masked language modeling: corrupt ~15% of tokens, run the full bidirectional stack, and reconstruct the originals at the corrupted positions with a cross-entropy loss summed only over those positions. The 80/10/10 recipe (mask / random / keep) exists to narrow the pretrain-vs-finetune mismatch caused by the artificial [MASK] token, pushing the model to contextualize every position rather than only obvious masks. Original BERT paired MLM with Next-Sentence Prediction, but NSP proved unhelpful and was dropped (or replaced by ALBERT\u0026rsquo;s harder Sentence-Order Prediction). The defining cost of MLM is signal density: only the masked fraction yields loss, so MLM is ~$6\\times$ less supervision-dense per forward pass than autoregressive training — the trade you accept in exchange for bidirectional representations.\nWe now have a learning signal but have glossed over how a top-layer vector becomes a vocabulary distribution, and how [CLS] becomes a label. Lesson 6 builds those heads — the MLM head with weight tying, and the pooling heads for sequence- and token-level prediction — and contrasts them with the decoder\u0026rsquo;s unembedding.\n← Previous: Lesson 4 — The Encoder Block \u0026amp; Stack · Course index · Next: Lesson 6 — From Representations to Predictions → Appendices: A — Encoder Family · B — Encoder vs. Decoder Cheat Sheet · C — Acronyms The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/tutorials/encoder-deep-dive/lesson-05-masked-language-modeling/","summary":"\u003ch1 id=\"lesson-5--masked-language-modeling\"\u003eLesson 5 — Masked Language Modeling\u003c/h1\u003e\n\u003cp\u003e\u003cem\u003eTechnical course on encoder LLM mechanics — Lesson 5 of 10.\u003c/em\u003e\u003c/p\u003e\n\u003chr\u003e\n\u003ch2 id=\"recap-of-lesson-4\"\u003eRecap of Lesson 4\u003c/h2\u003e\n\u003cp\u003eStacking $N$ bidirectional blocks turns the input $H^{(0)}$ into $H^{(N)}\\in\\mathbb R^{n\\times d}$, a contextual vector per token plus a \u003ccode\u003e[CLS]\u003c/code\u003e summary register — the encoder\u0026rsquo;s entire output, with no predictions attached. But why does the stack learn \u003cem\u003euseful\u003c/em\u003e representations? A randomly initialized encoder produces garbage; the representations become meaningful only because pretraining pushes a learning signal through every weight. This lesson derives that signal. It is the single biggest departure from the decoder, and Lesson 3 already told us why one is needed: bidirectionality kills next-token prediction.\u003c/p\u003e","title":"Masked Language Modeling"},{"content":"Lesson 5 — The Output Head \u0026amp; the Language-Modeling Objective Technical course on LLM mechanics — Lesson 5 of 10.\nRecap of Lesson 4 A block is two pre-norm residual sub-layers: multi-head attention and a position-wise MLP ($W_2\\phi(W_1\\mathbf x+\\mathbf b_1)+\\mathbf b_2$, $d_{ff}=4d$), the latter supplying nonlinear capacity and $\\approx\\tfrac23$ of the $P\\approx12Nd^2+2Vd$ parameters. Residuals give the layer Jacobian $I+\\partial\\mathcal F$ (identity gradient path → trainable depth); LayerNorm/RMSNorm standardize features with a learned gain. Stacking $N$ blocks yields $H^{(N)}\\in\\mathbb R^{n\\times d}$. We now map $H^{(N)}$ to next-token probabilities and state the training objective precisely.\nThis is where the forward pass finishes. We have a contextual vector per position; what we still lack is a prediction we can score. This lesson closes that gap twice over — it reads $H^{(N)}$ out into a probability distribution, and it pins down exactly what \u0026ldquo;good\u0026rdquo; means by writing the objective the model is trained against. That objective will carry us straight into the rest of the course.\n1. From hidden state to logits By the causal mask (Lesson 3), row $i$ of $H^{(N)}$, call it $\\mathbf h_i\\in\\mathbb R^d$, is a function of $x_{\\le i}$ only. The output head (often a final norm then a linear unembedding $W_U\\in\\mathbb R^{d\\times V}$) produces logits:\n$$ \\mathbf z_i=\\mathbf h_i^\\top W_U\\in\\mathbb R^V. $$$\\mathbf z_i$ is unnormalized — components in $(-\\infty,\\infty)$, one per vocabulary token. Because the causal mask makes every position predict its successor, a single forward pass over a length-$n$ sequence yields $n$ logit vectors $\\{\\mathbf z_i\\}_{i=1}^n$ — i.e. $n$ supervised prediction problems for the price of one pass. This parallelism over positions is what makes pretraining throughput viable.\nWeight tying A common choice sets $W_U=E^\\top$ (reuse the transposed embedding). Justification: $E:\\text{token}\\mapsto\\mathbb R^d$, and we want the inverse direction $\\mathbb R^d\\mapsto\\text{token}$; using the same matrix ties the two representations of each token and removes a $Vd$-parameter matrix (significant when $V$ is large). Whether tied or independent, $W_U$ is [LEARNED].\nLogits are scores, not probabilities — they can be any real number and they do not sum to anything in particular. To make them a distribution we can sample from and, crucially, score against the truth, we pass them through softmax.\n2. Softmax: logits to a categorical distribution $$ \\hat p_w=\\big[\\mathrm{softmax}(\\mathbf z)\\big]_w=\\frac{e^{z_w}}{\\sum_{m=1}^{V}e^{z_m}},\\qquad \\hat{\\mathbf p}\\in\\Delta^{V-1}. $$Two structural facts. (i) Shift invariance: $\\mathrm{softmax}(\\mathbf z+c\\mathbf 1)=\\mathrm{softmax}(\\mathbf z)$, so only logit differences matter; implementations subtract $\\max_m z_m$ before exponentiating for numerical stability (avoids overflow of $e^{z}$). (ii) Temperature: $\\mathrm{softmax}(\\mathbf z/T)$ interpolates between argmax ($T\\to0$) and uniform ($T\\to\\infty$) — the inference knob of Lesson 10.\nWorked micro-example, $V=3$, $\\mathbf z=[2.0,1.0,0.1]$: $e^{\\mathbf z}=[7.389,2.718,1.105]$, sum $11.213$, so $\\hat{\\mathbf p}=[0.659,0.242,0.099]$. The exponential maps a 1.0-logit gap into a $\\sim$2.7× probability ratio.\nFigure 1. Logits to probabilities, with the lesson\u0026rsquo;s three numbers.\n3. The objective: maximum likelihood = minimum cross-entropy We can now produce a distribution $\\hat{\\mathbf p}$ at every position. That makes the model a candidate, but it does not yet tell us whether it is a good one. Defining \u0026ldquo;good\u0026rdquo; is the pivot from architecture to training: we want $\\theta$ that makes the observed text as probable as possible.\nUsing the autoregressive factorization (Lesson 1), the log-likelihood of a sequence under $\\theta$ is\n$$ \\log P_\\theta(x_{1:n})=\\sum_{t=1}^{n}\\log P_\\theta(x_t\\mid x_{\\lt t})=\\sum_{t=1}^{n}\\log \\hat p_{\\,t,\\,x_t}, $$where $\\hat p_{t,x_t}$ is the probability the model assigned at position $t$ to the token that actually occurred. Maximum-likelihood estimation maximizes this over the corpus; equivalently (negate, average) we minimize the cross-entropy loss\n$$ \\mathcal L(\\theta)=-\\frac1n\\sum_{t=1}^n\\log \\hat p_{\\,t,\\,x_t}. $$(The full information-theoretic treatment is Lesson 6.) This objective is self-supervised: the label at position $t$ is simply $x_{t+1}$, already present in the data. No human annotation is required, which is what permits training on $10^{12}$+ tokens of raw text.\n4. The key gradient identity: $\\partial\\mathcal L/\\partial\\mathbf z=\\hat{\\mathbf p}-\\mathbf y$ We have a loss to minimize; minimizing it means knowing which way to nudge the logits that produced it. The derivative of the loss with respect to those logits turns out to be strikingly simple, and that simplicity is no accident — it is precisely why softmax and cross-entropy are chosen as a pair.\nThis single result is why softmax and cross-entropy are paired, and it seeds the entire backward pass (Lesson 7). Let $\\mathbf y$ be the one-hot target ($y_w=\\delta_{w,x_t}$) and $\\ell=-\\sum_w y_w\\log\\hat p_w=-\\log\\hat p_{x_t}$. Differentiate w.r.t. logit $z_k$, using the softmax Jacobian $\\partial\\hat p_w/\\partial z_k=\\hat p_w(\\delta_{wk}-\\hat p_k)$:\n$$ \\frac{\\partial \\ell}{\\partial z_k} =-\\sum_w \\frac{y_w}{\\hat p_w}\\,\\hat p_w(\\delta_{wk}-\\hat p_k) =-\\sum_w y_w(\\delta_{wk}-\\hat p_k) =\\hat p_k\\underbrace{\\textstyle\\sum_w y_w}_{=1}-y_k=\\hat p_k-y_k. $$Hence, in vector form,\n$$ \\boxed{\\ \\frac{\\partial\\mathcal L}{\\partial\\mathbf z}=\\hat{\\mathbf p}-\\mathbf y\\ } $$The gradient at the logits is just predicted minus target — the residual error. It is bounded, numerically clean, and avoids the $1/\\hat p$ blow-up that would arise if softmax and cross-entropy were differentiated separately. This is the elegant consequence of using cross-entropy with a softmax head (and the reason frameworks fuse them into one cross_entropy_with_logits op).\nFigure 2. The gradient at the logits is just predicted minus target.\n5. The complete forward map With the head and objective in place, every piece of $f$ assembled across Lessons 2–5 can be written as one chain. This is the whole forward pass, end to end:\n$$ \\text{tokens}\\xrightarrow{E(+PE)}H^{(0)}\\xrightarrow{N\\text{ blocks}}H^{(N)}\\xrightarrow{W_U}\\mathbf z\\xrightarrow{\\mathrm{softmax}}\\hat{\\mathbf p}. $$The full learned parameter set is\n$$ \\theta=\\Big\\{E,\\ \\{W_Q,W_K,W_V,W_O\\}_{\\ell=1}^N,\\ \\{W_1,W_2,\\mathbf b_1,\\mathbf b_2\\}_{\\ell=1}^N,\\ \\{\\gamma,\\beta\\}_{\\ell},\\ W_U\\Big\\}. $$Training adjusts all of it; inference holds all of it fixed and typically evaluates only the final position\u0026rsquo;s $\\hat{\\mathbf p}$.\nLabel smoothing (a [CONFIG] aside). Some training runs replace the one-hot $\\mathbf y$ with a softened target $\\mathbf y'=(1-\\varepsilon)\\mathbf y+\\tfrac{\\varepsilon}{V}\\mathbf 1$. This caps the achievable confidence (preventing logits from diverging to $\\pm\\infty$) and slightly improves calibration. With a soft target the cross-entropy sum no longer collapses to a single term — all $V$ terms survive — which is why Lesson 6 presents the general form.\nLearned / config / state Quantity Category Unembedding $W_U$ (often $=E^\\top$) [LEARNED] Softmax; vocab size $V$ [CONFIG] Label-smoothing $\\varepsilon$ (if used) [CONFIG] Logits $\\mathbf z$, probabilities $\\hat{\\mathbf p}$ [STATE] Figure 3. The complete forward map, Lessons 2 through 5.\nConclusion The output head maps each contextual vector $\\mathbf h_i$ to logits $\\mathbf z_i=\\mathbf h_i^\\top W_U$ (optionally tying $W_U=E^\\top$ to save $Vd$ parameters), and softmax — shift-invariant, numerically stabilized by max-subtraction — turns them into a categorical distribution. Causal masking gives $n$ training signals per forward pass. The objective is maximum likelihood of the observed tokens, equivalently minimum cross-entropy, and it is self-supervised because the target at $t$ is just $x_{t+1}$. The pairing of softmax with cross-entropy yields the clean, bounded gradient $\\partial\\mathcal L/\\partial\\mathbf z=\\hat{\\mathbf p}-\\mathbf y$ (predicted minus target), which is the seed of backpropagation. Learned: $W_U$. Config: vocab, softmax, optional label smoothing. State: logits and probabilities.\nThe forward pass is now complete and we have named the quantity it is judged by. Before we can minimize that loss, it pays to understand what it actually measures. Next we develop the loss itself information-theoretically — cross-entropy as entropy plus KL divergence — and its human-facing form, perplexity.\n← Lesson 4 — The Transformer Block · Course index · Next: Lesson 6 — The Loss Function → Appendices: A — Acronyms · B — Formulas · C — Attention Variants The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/tutorials/llm-lessons/lesson-05-output-head-and-objective/","summary":"\u003ch1 id=\"lesson-5--the-output-head--the-language-modeling-objective\"\u003eLesson 5 — The Output Head \u0026amp; the Language-Modeling Objective\u003c/h1\u003e\n\u003cp\u003e\u003cem\u003eTechnical course on LLM mechanics — Lesson 5 of 10.\u003c/em\u003e\u003c/p\u003e\n\u003chr\u003e\n\u003ch2 id=\"recap-of-lesson-4\"\u003eRecap of Lesson 4\u003c/h2\u003e\n\u003cp\u003eA block is two pre-norm residual sub-layers: multi-head attention and a position-wise MLP ($W_2\\phi(W_1\\mathbf x+\\mathbf b_1)+\\mathbf b_2$, $d_{ff}=4d$), the latter supplying nonlinear capacity and $\\approx\\tfrac23$ of the $P\\approx12Nd^2+2Vd$ parameters. Residuals give the layer Jacobian $I+\\partial\\mathcal F$ (identity gradient path → trainable depth); LayerNorm/RMSNorm standardize features with a learned gain. Stacking $N$ blocks yields $H^{(N)}\\in\\mathbb R^{n\\times d}$. We now map $H^{(N)}$ to next-token probabilities and state the training objective precisely.\u003c/p\u003e","title":"Output Head and Objective"},{"content":"Lesson 5 — The Encoder–Decoder Block \u0026amp; Full Architecture Technical course on encoder–decoder transformers — Lesson 5 of 10.\nRecap of Lesson 4 Cross-attention is scaled dot-product attention with queries from the decoder and keys/values from the encoder memory: $Q=H_{\\text{dec}}W_Q^c$, $K,V=H_{\\text{enc}}W_K^c,H_{\\text{enc}}W_V^c$, giving a rectangular alignment $A\\in\\mathbb R^{m\\times n}$ and output $AV\\in\\mathbb R^{m\\times d_v}$. It uses the source padding mask but no causal mask, and it is the sole bridge for the training gradient into the encoder. With all three decoder sublayers derived, we now assemble the full model, count its parameters, and locate where T5 and BART deviate from the textbook Transformer.\n1. The two block types, side by side The architecture is built from two block types repeated $N_{\\text{enc}}$ and $N_{\\text{dec}}$ times. Writing $\\mathrm{Norm}$ for the (pre-)normalization:\n$$ \\textbf{Encoder block:}\\quad \\begin{aligned} H' \u0026= H + \\mathrm{SelfAttn}_{\\text{bidir}}(\\mathrm{Norm}(H))\\\\ H^{\\text{out}} \u0026= H' + \\mathrm{FFN}(\\mathrm{Norm}(H')) \\end{aligned} $$$$ \\textbf{Decoder block:}\\quad \\begin{aligned} H_1 \u0026= H + \\mathrm{SelfAttn}_{\\text{causal}}(\\mathrm{Norm}(H))\\\\ H_2 \u0026= H_1 + \\mathrm{CrossAttn}(\\mathrm{Norm}(H_1),\\,H_{\\text{enc}})\\\\ H_3 \u0026= H_2 + \\mathrm{FFN}(\\mathrm{Norm}(H_2)) \\end{aligned} $$The encoder block has two sublayers; the decoder block has three. Every decoder block consumes the same $H_{\\text{enc}}$ — the encoder is run once, and its output is broadcast to all $N_{\\text{dec}}$ cross-attention sublayers. This is the structural picture of the course invariant: one compression ($H_{\\text{enc}}$), many conditioned generations.\nFigure 1. Two block types, repeated.\n2. The full forward map End to end, an encoder–decoder transformer computes:\n$$ x \\xrightarrow{E_x(+\\text{pos})} H^{(0)}_{\\text{enc}} \\xrightarrow{N_{\\text{enc}}\\text{ enc blocks}} H_{\\text{enc}} $$$$ y_{\\lt t} \\xrightarrow{E_y(+\\text{pos})} H^{(0)}_{\\text{dec}} \\xrightarrow[\\text{each block also reads }H_{\\text{enc}}]{N_{\\text{dec}}\\text{ dec blocks}} H_{\\text{dec}} \\xrightarrow{W_U} \\mathbf z_t \\xrightarrow{\\mathrm{softmax}} \\hat{\\mathbf p}_t . $$flowchart TB subgraph ENC[\"Encoder ×N_enc (bidirectional)\"] x[\"x tokens\"] --\u003e e0[\"embed + pos\"] --\u003e eb[\"[ self-attn → FFN ] × N_enc\"] --\u003e Henc[\"H_enc (n × d)\"] end subgraph DEC[\"Decoder ×N_dec (causal)\"] y[\"y_","permalink":"https://www.fanfani.net/tutorials/encoder-decoder-lessons/lesson-05-the-encoder-decoder-block/","summary":"\u003ch1 id=\"lesson-5--the-encoderdecoder-block--full-architecture\"\u003eLesson 5 — The Encoder–Decoder Block \u0026amp; Full Architecture\u003c/h1\u003e\n\u003cp\u003e\u003cem\u003eTechnical course on encoder–decoder transformers — Lesson 5 of 10.\u003c/em\u003e\u003c/p\u003e\n\u003chr\u003e\n\u003ch2 id=\"recap-of-lesson-4\"\u003eRecap of Lesson 4\u003c/h2\u003e\n\u003cp\u003eCross-attention is scaled dot-product attention with queries from the decoder and keys/values from the encoder memory: $Q=H_{\\text{dec}}W_Q^c$, $K,V=H_{\\text{enc}}W_K^c,H_{\\text{enc}}W_V^c$, giving a rectangular alignment $A\\in\\mathbb R^{m\\times n}$ and output $AV\\in\\mathbb R^{m\\times d_v}$. It uses the source padding mask but no causal mask, and it is the sole bridge for the training gradient into the encoder. With all three decoder sublayers derived, we now assemble the full model, count its parameters, and locate where T5 and BART deviate from the textbook Transformer.\u003c/p\u003e","title":"The Encoder–Decoder Block \u0026 Full Architecture"},{"content":"Chapter 5 — The Readers: Encoder Models The storyteller of the last chapter is built to write. But writing is not the only thing one might want from a language machine. Often you already have the text — a review, an email, a legal clause, a search query — and what you want is not more text but understanding: is this review positive or negative, is this email spam, which words here are people\u0026rsquo;s names, does this query mean the same thing as that document. For jobs like these, the machinery from Part I is arranged into a different shape entirely. These are encoder models, and the most famous of them is BERT. If the decoder is a writer, the encoder is a reader.\nReaders get to look both ways The single most important difference between a reader and a writer is what each is allowed to look at. A writer, as we saw, must work strictly left to right, blind to anything ahead, because its training game forbids peeking at the answer. A reader is under no such constraint. When an encoder processes a word, it can look in both directions at once — at the words before it and the words after it.\nThis freedom is not a minor tweak; it is the whole point, and it changes everything downstream. Think about how much the meaning of a word can depend on what comes later. In \u0026ldquo;we sat on the bank of the river,\u0026rdquo; the word \u0026ldquo;bank\u0026rdquo; only resolves into \u0026ldquo;riverbank\u0026rdquo; once you reach \u0026ldquo;river,\u0026rdquo; which arrives afterward. A strictly left-to-right reader would have to commit to a meaning for \u0026ldquo;bank\u0026rdquo; before seeing the word that settles it. The bidirectional reader simply waits, takes in the whole sentence, and lets every word be informed by its complete surroundings.\nA homely analogy: the writer is solving a crossword strictly top to bottom, never permitted to glance at the clues below. The reader fills in the whole grid with every clue visible at once, freely using answers on the right to help with answers on the left. For the job of understanding, having every clue in view is an enormous advantage.\nFigure 1. The crossword, solved two ways.\nWhat a reader actually produces This is the part that trips people up, so it is worth stating plainly: an encoder does not produce text. It does not answer in words. What it produces, for each word you feed it, is a rich bundle of numbers — a contextual representation, which we can think of as a meaning-fingerprint — that has been shaped by the entire sentence around that word.\nSo the word \u0026ldquo;bank\u0026rdquo; in the river sentence comes out with a completely different fingerprint than \u0026ldquo;bank\u0026rdquo; in \u0026ldquo;I deposited the check at the bank.\u0026rdquo; Same spelling, same starting point on the map of meaning, but utterly different fingerprints once the surrounding context has done its work. That is the encoder\u0026rsquo;s entire product: not a dictionary definition but a context-aware reading of what each word means right here, in this particular sentence. A plain dictionary gives one fixed entry for \u0026ldquo;bank\u0026rdquo;; a skilled human reader instead infers which sense you meant from the sentence. The encoder is built to behave like the skilled reader, and the meaning-fingerprints are what it hands off for someone else to use.\nThe marker tokens that frame every input Encoders wrap every input in a few special marker symbols that carry no word meaning of their own but do important structural work. Two are worth knowing.\nThe first sits at the very front of every input — think of it as a blank summary card stapled to the front of the document. Because the reader can see everything at once, this card gradually fills up, as the model reads, with a summary of the whole text. Later, when we want a single overall verdict about the entire input — is this spam, what is the topic, is the sentiment positive — we simply read that summary card. It gives the model a dedicated place to accumulate a sense of the whole.\nThe second is a divider, like a staple between two stapled documents, used to separate one sentence from another. It lets an encoder handle pairs of inputs at once, which is exactly what you need for questions like \u0026ldquo;do these two sentences mean the same thing?\u0026rdquo; or \u0026ldquo;does this passage answer this question?\u0026rdquo; A third marker, the blank, appears only during training, and it leads us to the cleverest part of the whole story.\nHow do you train a reader? Fill in the blanks Here is a real puzzle. A writer trains by guessing the next word, billions of times. A reader cannot play that game — it can see in both directions, so the \u0026ldquo;next word\u0026rdquo; is sitting right there in plain view, and guessing it would be cheating that teaches nothing. So how does a reader learn anything from raw text?\nThe answer is a fill-in-the-blank game. Take a sentence, cover up about one word in seven at random, and train the model to recover the hidden words from everything around them. \u0026ldquo;The ___ sat on the mat\u0026rdquo; — the model should produce \u0026ldquo;cat.\u0026rdquo; Crucially, the model never sees the word it is trying to guess, so its ability to look both left and right is now a genuine, honest advantage rather than a cheat. It is the language-class cloze exercise, or a game of Mad Libs run in reverse.\nThis is the deep reason the two model families differ at the root. Bidirectionality and next-word prediction are fundamentally incompatible: you can let a model see in all directions, or you can train it to predict the next word, but you cannot do both, because seeing everything makes next-word prediction trivial. Choosing to build a reader that sees everything forces you to train it on a fill-in-the-blank objective instead. The reading ability and the fill-in-the-blank game are two sides of one coin.\nThere is a clever wrinkle in the recipe. If the model only ever had to think hard at the blank symbol, it would grow lazy everywhere else — but in real use there are no blanks, just ordinary complete text. So of the words chosen to be quizzed, most are blanked out, a small share are swapped for a random wrong word, and a small share are left exactly as they are while still being quizzed. The wrong-word case teaches the model not to blindly trust the word it sees, since it might be a plant. The leave-it-alone case keeps the model paying attention even to perfectly normal words, which is the situation it will always face once deployed. The effect is a reader that stays alert across the entire text, not just at the obvious gaps.\nFigure 2. Fill in the blanks, and the wrinkle that keeps the reader alert.\nTurning fingerprints into answers The encoder hands off meaning-fingerprints, but a fingerprint is not yet an answer. To get an answer you bolt on a small attachment — a head — tailored to the shape of the question. The beauty is that the same understanding, captured once in the fingerprints, can be pointed at very different tasks just by changing the little head on top.\nIf you want one overall label for the whole text — positive or negative, spam or not — you read the summary card and run it through a tiny classifier. If you want a label for every word — which words are names, places, dates — you run a small classifier on each word\u0026rsquo;s fingerprint in turn. If you want to point at where in a passage the answer to a question lives, you have the head mark a start word and an end word, highlighting the answer span. The reader\u0026rsquo;s outputs are clean and bounded in a way a writer\u0026rsquo;s free-form text is not: a tidy probability across a fixed set of categories, or a highlighted span guaranteed to be real text from the document. It cannot wander off and invent a category that doesn\u0026rsquo;t exist. For high-volume, sensitive jobs like moderating content or tagging billions of documents, those tidy, trustworthy outputs are both cheaper and safer than asking a storyteller to write its verdict in prose.\nFigure 3. Three little heads, one set of fingerprints.\nFast, single-pass, and unable to write A reader has none of the slow word-by-word loop of a writer. It takes in the whole input in a single pass and is immediately done — there is no generating, no appending, no sampling. This is why encoders are cheap and quick as analysis tools. It is also exactly why they fundamentally cannot produce text: there is no mechanism in them for extending an output one word at a time. If you want generation, you must attach an actual writer. A reader understands; it does not speak.\nThis also changes how you grade one. A writer can be measured by how unsurprised it is by the next word, on average — but that yardstick depends on the left-to-right, next-word structure a reader doesn\u0026rsquo;t have, so it simply doesn\u0026rsquo;t apply. Instead, readers are judged by what they enable: attach a head, train it briefly on a real task, and see how well it scores on standard batteries of understanding problems. A reader\u0026rsquo;s worth is measured by how good its understanding turns out to be once you put it to work.\nA lesson worth remembering: BERT was undertrained One of the most instructive episodes in this corner of the field is the story of RoBERTa. Researchers took BERT\u0026rsquo;s exact design — not one structural change — and simply trained it far better: much more data, much longer, with a fresh random set of blanks each time and other refinements. The result was a large jump in quality. The startling conclusion was that the original BERT had not been designed poorly; it had merely been undertrained. Sometimes the biggest gains come not from a cleverer architecture but from feeding and training an existing one properly. It is a useful corrective to the assumption that progress always means a new design.\nWhy readers still matter, and how they cooperate with writers It is tempting to think that, in the age of chatbots that can do anything, the humble reader is obsolete. The opposite is true. Encoders are the quiet, high-throughput workhorses behind a great deal of what happens on the internet: search, content moderation, classification, tagging, deduplication. They are chosen precisely because they run in a single cheap pass and produce clean, structured, trustworthy outputs.\nTheir biggest modern role is turning whole sentences or documents into a single meaning-fingerprint — an embedding — so that texts with similar meaning land near each other in a kind of meaning-space. This powers semantic search: finding documents that mean the same thing as your query, not merely share its keywords. And it underlies the most important pattern in applied AI today, often called retrieval-augmented generation, in which the two families cooperate beautifully. A reader, fast and search-native, finds the handful of documents most relevant to your question; a writer then composes an answer using them. The reader retrieves; the writer generates. Each does exactly what its shape was built for. Far from competing, the reader and the writer turn out to be natural partners.\nWhat to carry forward An encoder is a reader. It looks in both directions at once, which makes it superb at understanding but means it can never write — and which forces its training to be a fill-in-the-blank game rather than next-word prediction. Its product is a set of context-aware meaning-fingerprints, turned into clean, bounded answers by small task-specific heads. Readers are the cheap, fast, trustworthy workhorses behind search, classification, and the retrieval half of modern AI systems.\nWe have now met the writer and the reader. The obvious next question is: what if you need both — to read one thing thoroughly and then write a different thing based on it, as in translation or summarization? That calls for bolting a reader and a writer together, which is the subject of Chapter 6 .\n← Chapter 4 · The Storytellers · Contents · Chapter 6 · The Translators → The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/tutorials/inside-the-machines/chapter-05-encoder-models/","summary":"\u003ch1 id=\"chapter-5--the-readers-encoder-models\"\u003eChapter 5 — The Readers: Encoder Models\u003c/h1\u003e\n\u003cp\u003eThe storyteller of the last chapter is built to write. But writing is not the only thing one might want from a language machine. Often you already have the text — a review, an email, a legal clause, a search query — and what you want is not more text but \u003cem\u003eunderstanding\u003c/em\u003e: is this review positive or negative, is this email spam, which words here are people\u0026rsquo;s names, does this query mean the same thing as that document. For jobs like these, the machinery from Part I is arranged into a different shape entirely. These are \u003cem\u003eencoder models\u003c/em\u003e, and the most famous of them is BERT. If the decoder is a writer, the encoder is a reader.\u003c/p\u003e","title":"The Readers: Encoder Models"},{"content":"Lesson 5 — The Simplified Training Objective Recap of Lesson 4. We maximize a variational lower bound $L = L_T + \\sum_{t\\ge2}L_{t-1} + L_0$. The prior term $L_T$ is a parameter-free constant; the reconstruction term $L_0$ is minor; the denoising terms $L_{t-1}$ carry the learning. With fixed reverse variance $\\sigma_t^2 I$, each reduces to a weighted squared error between the network\u0026rsquo;s reverse mean $\\mu_\\theta(x_t,t)$ and the analytic posterior mean $\\tilde\\mu_t(x_t,x_0)$: $L_{t-1}=\\frac{1}{2\\sigma_t^2}\\|\\tilde\\mu_t-\\mu_\\theta\\|^2 + C$. Now we collapse this into one clean loss.\nBy now we have assembled a fair amount of apparatus: a variational bound, a tractable posterior, a KL between Gaussians, a schedule of fiddly coefficients. It would be reasonable to expect the training objective to look like all of that machinery bolted together. Instead, when the right substitution is made, the apparatus folds in on itself and almost everything cancels — leaving a single line you could explain to someone who has never heard the word \u0026ldquo;posterior\u0026rdquo;: look at a noisy image, guess the noise. This lesson is the story of that collapse, and of why the messy weight that drops out is worth throwing away.\nOverview Lesson 4 left the objective in mean-matching form: the regression target $\\tilde\\mu_t(x_t,x_0)$ from Lesson 3 is a schedule-dependent affine blend of $x_0$ and $x_t$. Asking a network to hit that exact blend couples the optimizer\u0026rsquo;s target to the noise schedule and rescales it across $t$. The remedy is a change of parameterization. Writing $\\tilde\\mu_t$ in its noise form and giving $\\mu_\\theta$ the identical functional shape causes the schedule factors to cancel algebraically, leaving a plain $\\ell_2$ regression onto the injected noise $\\epsilon$. The DDPM observation is that discarding the per-timestep weight $w_t$ that falls out of the ELBO further improves perceptual sample quality. We make the cancellation explicit, derive $w_t$ exactly, locate the resulting loss in the space of SNR-weighted objectives, and lay out the linear correspondence between $\\epsilon$-, $x_0$-, and $v$-targets.\nParameterizing the mean through noise From Lesson 3, the posterior mean in noise form is\n$$ \\tilde\\mu_t(x_t, x_0) = \\frac{1}{\\sqrt{\\alpha_t}}\\left(x_t - \\frac{\\beta_t}{\\sqrt{1-\\bar\\alpha_t}}\\,\\epsilon\\right), \\qquad \\text{where } x_t = \\sqrt{\\bar\\alpha_t}x_0 + \\sqrt{1-\\bar\\alpha_t}\\,\\epsilon. $$We choose to give the network the same shape, but with a predicted noise $\\epsilon_\\theta(x_t,t)$ in place of the true $\\epsilon$:\n$$ \\boxed{\\ \\mu_\\theta(x_t, t) = \\frac{1}{\\sqrt{\\alpha_t}}\\left(x_t - \\frac{\\beta_t}{\\sqrt{1-\\bar\\alpha_t}}\\,\\epsilon_\\theta(x_t,t)\\right)\\ } $$This is the $\\epsilon$-parameterization. The network\u0026rsquo;s only job is to look at the noisy $x_t$ (and the timestep $t$) and predict the noise vector $\\epsilon$ that contaminated it. Note that this is not the only admissible reparameterization — it is a deliberate one, chosen precisely because it cancels the schedule (below) and because it makes the regression target unit-variance for all $t$ (see Lesson 9, score matching).\nThe cancellation, in full Watch closely now; this is the trick the whole lesson turns on.\nSubstitute both means into the denoising term $L_{t-1}=\\frac{1}{2\\sigma_t^2}\\|\\tilde\\mu_t-\\mu_\\theta\\|^2$. Write $c_t := 1/\\sqrt{\\alpha_t}$ and $b_t := \\beta_t/\\sqrt{1-\\bar\\alpha_t}$ so that $\\tilde\\mu_t = c_t(x_t - b_t\\,\\epsilon)$ and $\\mu_\\theta = c_t(x_t - b_t\\,\\epsilon_\\theta)$. Then\n$$ \\tilde\\mu_t - \\mu_\\theta = c_t\\big[(x_t - b_t\\epsilon) - (x_t - b_t\\epsilon_\\theta)\\big] = c_t\\,b_t\\,(\\epsilon_\\theta - \\epsilon) = \\frac{1}{\\sqrt{\\alpha_t}}\\cdot\\frac{\\beta_t}{\\sqrt{1-\\bar\\alpha_t}}\\,(\\epsilon_\\theta - \\epsilon). $$The $x_t$ pieces — the only $x_0$- and schedule-coupled part of each mean — are identical and subtract away exactly. Taking the squared norm,\n$$ \\|\\tilde\\mu_t - \\mu_\\theta\\|^2 = (c_t b_t)^2\\,\\|\\epsilon - \\epsilon_\\theta\\|^2 = \\frac{\\beta_t^2}{\\alpha_t(1-\\bar\\alpha_t)}\\,\\|\\epsilon - \\epsilon_\\theta\\|^2, $$and dividing by $2\\sigma_t^2$,\n$$ L_{t-1} = \\frac{1}{2\\sigma_t^2}\\cdot\\frac{\\beta_t^2}{\\alpha_t(1-\\bar\\alpha_t)}\\,\\big\\|\\epsilon - \\epsilon_\\theta(x_t,t)\\big\\|^2 = w_t\\,\\big\\|\\epsilon - \\epsilon_\\theta(x_t,t)\\big\\|^2, $$which defines the exact per-timestep weight\n$$ \\boxed{\\ w_t = \\frac{\\beta_t^2}{2\\sigma_t^2\\,\\alpha_t(1-\\bar\\alpha_t)}\\ }. $$Substituting $x_t = \\sqrt{\\bar\\alpha_t}x_0 + \\sqrt{1-\\bar\\alpha_t}\\,\\epsilon$ gives the fully explicit ELBO denoising term:\n$$ \\boxed{\\ L_{t-1} = \\frac{\\beta_t^2}{2\\sigma_t^2\\,\\alpha_t(1-\\bar\\alpha_t)}\\, \\Big\\|\\,\\epsilon - \\epsilon_\\theta\\big(\\sqrt{\\bar\\alpha_t}\\,x_0 + \\sqrt{1-\\bar\\alpha_t}\\,\\epsilon,\\ t\\big)\\,\\Big\\|^2\\ } $$The KL between two Gaussian reverse steps has become a squared error between the true noise and the predicted noise, with a known, closed-form coefficient. Probability theory in, regression out.\nRemark (the two standard variance choices). DDPM fixes $\\sigma_t^2$ to one of two endpoints of the reverse-variance interval: $\\sigma_t^2=\\beta_t$ (upper bound, matching $q(x_t\\mid x_{t-1})$) or $\\sigma_t^2=\\tilde\\beta_t=\\frac{1-\\bar\\alpha_{t-1}}{1-\\bar\\alpha_t}\\beta_t$ (lower bound, the true posterior variance). Both are valid; the choice rescales $w_t$ but not the optimum. With $\\sigma_t^2=\\tilde\\beta_t$ one obtains $w_t=\\frac{\\beta_t^2}{2\\tilde\\beta_t\\,\\alpha_t(1-\\bar\\alpha_t)}$; the two agree to leading order for small $\\beta_t$. Either way the optimizer\u0026rsquo;s fixed point is the conditional mean $\\mathbb E[\\epsilon\\mid x_t,t]$, independent of $w_t\\gt 0$, since $w_t$ scales the gradient but not its zero.\nFigure 1. The great cancellation, in four moves.\nWeights and signal-to-noise ratio Define the per-timestep signal-to-noise ratio $\\mathrm{SNR}(t)=\\bar\\alpha_t/(1-\\bar\\alpha_t)$, the ratio of signal power to noise power in $x_t=\\sqrt{\\bar\\alpha_t}x_0+\\sqrt{1-\\bar\\alpha_t}\\,\\epsilon$. The variational weight is naturally expressed in these terms. With $\\sigma_t^2=\\beta_t$,\n$$ w_t=\\frac{\\beta_t^2}{2\\beta_t\\,\\alpha_t(1-\\bar\\alpha_t)}=\\frac{\\beta_t}{2\\alpha_t(1-\\bar\\alpha_t)}. $$For the variance-preserving schedule, Kingma et al. (2021) show the continuous-time ELBO contribution of each noise level is proportional to $-\\tfrac{d}{dt}\\mathrm{SNR}(t)$ times the $\\epsilon$-MSE: the loss is an SNR-weighted denoising score-matching objective, and the discrete $w_t$ above is its finite-difference image. Concretely, $w_t$ is large at small $t$ (high SNR, easy denoising) and small at large $t$ (low SNR, where $x_t$ is nearly pure noise).\nFigure 2. The weight the theory hands us, and why it gets thrown away.\nDropping the weights: the \u0026ldquo;simple\u0026rdquo; loss The exact objective is in hand. What follows is the small heresy that made diffusion work in practice — discarding the very weight the theory just handed us.\nDDPM\u0026rsquo;s celebrated empirical finding is that setting all weights to 1 trains better. Replacing $w_t$ by $1$ down-weights the high-SNR small-$t$ terms — which the ELBO weight inflates but which contribute imperceptible high-frequency detail — and relatively up-weights the structurally important mid/low-SNR terms. The result is the simplified objective:\n$$ \\boxed{\\ L_{\\text{simple}}(\\theta) = \\mathbb{E}_{t\\sim\\mathcal U\\{1,T\\},\\ x_0\\sim q,\\ \\epsilon\\sim\\mathcal N(0,I)}\\Big[\\big\\|\\,\\epsilon - \\epsilon_\\theta(\\sqrt{\\bar\\alpha_t}\\,x_0 + \\sqrt{1-\\bar\\alpha_t}\\,\\epsilon,\\ t)\\,\\big\\|^2\\Big]\\ } $$It compresses the entire training procedure into one expectation:\nDraw a real image $x_0\\sim q$. Draw a timestep $t\\sim\\mathcal U\\{1,\\dots,T\\}$. Draw noise $\\epsilon\\sim\\mathcal N(0,I)$. Form $x_t=\\sqrt{\\bar\\alpha_t}x_0+\\sqrt{1-\\bar\\alpha_t}\\,\\epsilon$ in one shot (Lesson 2\u0026rsquo;s closed-form marginal). Predict $\\epsilon_\\theta(x_t,t)$. Penalize $\\|\\epsilon-\\epsilon_\\theta\\|^2$; backpropagate; repeat. No adversary, no sequential rollout, no intractable normalizer — a regression problem with infinite, free-to-generate training pairs. Each $(x_0,t,\\epsilon)$ triple yields a fresh $(x_t,\\epsilon)$ example, and the unbiased single-sample Monte-Carlo estimate of the expectation is exactly the per-minibatch loss.\nRemark (likelihood vs. quality; min-SNR-$\\gamma$). $L_{\\text{simple}}$ is not the ELBO: maximizing it does not maximize $\\log p_\\theta(x_0)$. It trades likelihood (which the true weighting $w_t$ optimizes) for FID/perceptual quality. The two regimes can be reconciled by choosing a weighting $\\lambda_t$ between the extremes. Min-SNR-$\\gamma$ (Hang et al., 2023) sets $\\lambda_t=\\min\\{\\mathrm{SNR}(t),\\gamma\\}/\\mathrm{SNR}(t)$ in $x_0$-space (typically $\\gamma=5$), which clamps the otherwise unbounded high-SNR contribution and accelerates convergence; P2 weighting (Choi et al., 2022) uses $\\lambda_t=(k+\\mathrm{SNR}(t))^{-\\gamma}$ to concentrate capacity on the \u0026ldquo;content-bearing\u0026rdquo; middle noise levels. $L_{\\text{simple}}$ ($\\lambda_t\\equiv 1$ in $\\epsilon$-space), the ELBO ($\\lambda_t=w_t$), Min-SNR, and P2 are all instances of the same family $\\sum_t \\lambda_t\\,\\mathbb E\\|\\epsilon-\\epsilon_\\theta\\|^2$; only the per-$t$ emphasis differs. Since each is a positive reweighting of the same nonnegative per-$t$ MSE, all share the same global minimizer $\\epsilon_\\theta(x_t,t)=\\mathbb E[\\epsilon\\mid x_t,t]$; the weighting governs optimization dynamics and the implicit objective under finite capacity, not the idealized optimum.\nWhy predicting noise (not the image) works so well Scale stability. $\\epsilon$ has unit variance at every $t$, so the regression target is consistently scaled across timesteps — gentle on the optimizer and on weight-decay/learning-rate choices. The $x_0$-target, by contrast, is implicitly divided by $\\sqrt{1-\\bar\\alpha_t}$ relative to the $\\epsilon$-target (see the linear maps below), so its effective training-target variance blows up as $\\bar\\alpha_t\\to 1$ (small $t$) and collapses as $\\bar\\alpha_t\\to 0$ (large $t$). Connection to denoising score matching. As shown in Lesson 9, $\\epsilon_\\theta(x_t,t)\\approx -\\sqrt{1-\\bar\\alpha_t}\\,\\nabla_{x_t}\\log q(x_t)$, i.e. predicting noise is (up to the schedule factor) estimating the score. $L_{\\text{simple}}$ is thus a weighted denoising-score-matching objective. Empirics. It produces sharper, better samples than the $x_0$- or $\\mu$-targets in the original DDPM study. Equivalent targets, same model: the exact linear maps Given $x_t$ and $t$, the quantities $x_0$, $\\epsilon$, and the velocity $v$ are mutually affine. Fix $\\sqrt{\\bar\\alpha_t}$ and $\\sqrt{1-\\bar\\alpha_t}$. From the forward marginal,\n$$ x_0 = \\frac{x_t-\\sqrt{1-\\bar\\alpha_t}\\,\\epsilon}{\\sqrt{\\bar\\alpha_t}}, \\qquad \\epsilon = \\frac{x_t-\\sqrt{\\bar\\alpha_t}\\,x_0}{\\sqrt{1-\\bar\\alpha_t}}. $$The $v$-parameterization (Salimans \u0026amp; Ho, 2022) defines\n$$ \\boxed{\\ v := \\sqrt{\\bar\\alpha_t}\\,\\epsilon - \\sqrt{1-\\bar\\alpha_t}\\,x_0\\ } $$which inverts (using $\\bar\\alpha_t+(1-\\bar\\alpha_t)=1$) to\n$$ \\epsilon = \\sqrt{1-\\bar\\alpha_t}\\,x_t + \\sqrt{\\bar\\alpha_t}\\,v, \\qquad x_0 = \\sqrt{\\bar\\alpha_t}\\,x_t - \\sqrt{1-\\bar\\alpha_t}\\,v. $$Equivalently, with angle $\\phi_t$ such that $\\cos\\phi_t=\\sqrt{\\bar\\alpha_t}$, $\\sin\\phi_t=\\sqrt{1-\\bar\\alpha_t}$, the map $(x_0,\\epsilon)\\mapsto(x_t,v)$ is a rotation by $\\phi_t$; $v$ is the velocity $dx_t/d\\phi_t$ along the diffusion arc, and is the target of choice for stability near both endpoints and for distillation. Predicting any one target and converting to another at sampling time is exact:\n$$ \\hat x_0 = \\frac{1}{\\sqrt{\\bar\\alpha_t}}\\big(x_t - \\sqrt{1-\\bar\\alpha_t}\\,\\hat\\epsilon\\big), \\qquad \\hat\\epsilon = \\frac{x_t - \\sqrt{\\bar\\alpha_t}\\,\\hat x_0}{\\sqrt{1-\\bar\\alpha_t}}. $$ Remark (target-variance scaling). Under $x_t=\\cos\\phi_t\\,x_0+\\sin\\phi_t\\,\\epsilon$, the three targets have marginal scales $\\mathrm{Var}(\\epsilon)=1$ (constant), $\\mathrm{Var}(x_0)$ fixed by the data, and $\\mathrm{Var}(v)=\\cos^2\\phi_t\\cdot 1+\\sin^2\\phi_t\\cdot\\mathrm{Var}(x_0)$, which interpolates smoothly between the two endpoints. This is why $\\epsilon$-prediction is well-conditioned at high noise, $x_0$-prediction at low noise, and $v$-prediction across the whole range — a fact that the EDM preconditioning (Lesson 6/Karras et al., 2022) formalizes by scaling inputs, targets, and skip weights so the network always sees a unit-variance regression problem.\nFigure 3. Three targets, mutually affine given x_t.\nCode: the simple loss with optional SNR weighting import torch import torch.nn.functional as F def diffusion_loss(model, x0, alpha_bar, weighting=\u0026#34;simple\u0026#34;, gamma=5.0): \u0026#34;\u0026#34;\u0026#34; model: eps_theta(x_t, t) -\u0026gt; predicted noise, shape (B, C, H, W) x0: clean batch, shape (B, C, H, W) alpha_bar: precomputed cumulative products, shape (T,) on x0.device weighting: \u0026#34;simple\u0026#34; (lambda_t = 1), \u0026#34;snr\u0026#34;, or \u0026#34;min_snr\u0026#34; \u0026#34;\u0026#34;\u0026#34; B = x0.shape[0] T = alpha_bar.shape[0] t = torch.randint(0, T, (B,), device=x0.device) # (B,) ab = alpha_bar[t].view(B, 1, 1, 1) # (B,1,1,1) broadcast eps = torch.randn_like(x0) # (B,C,H,W) x_t = ab.sqrt() * x0 + (1.0 - ab).sqrt() * eps # forward marginal, one shot eps_pred = model(x_t, t) # (B,C,H,W) # per-example MSE, averaged over C,H,W -\u0026gt; shape (B,) se = F.mse_loss(eps_pred, eps, reduction=\u0026#34;none\u0026#34;).flatten(1).mean(1) if weighting == \u0026#34;simple\u0026#34;: w = torch.ones_like(se) else: snr = ab / (1.0 - ab) # (B,1,1,1) snr = snr.flatten() # (B,) if weighting == \u0026#34;snr\u0026#34;: w = snr # ELBO-like (eps-space ~ const; x0-space ~ snr) elif weighting == \u0026#34;min_snr\u0026#34;: w = torch.clamp(snr, max=gamma) / snr # Min-SNR-gamma in eps-space w = w.detach() # weighting must not receive gradient return (w * se).mean() Numerical notes. Precompute alpha_bar in float64 then cast, since $\\bar\\alpha_t=\\prod_{s\\le t}(1-\\beta_s)$ underflows toward $0$ at large $t$ for $T\\sim 1000$. Clamp $1-\\bar\\alpha_t$ away from $0$ (e.g. $\\ge 10^{-5}$) before .sqrt() at $t=0$ to avoid division by zero in $\\hat x_0$. Detach the weights so gradients flow only through eps_pred. The reduction=\u0026quot;none\u0026quot; + per-example mean keeps the loss scale independent of resolution.\nExercises Throughout, use the lesson\u0026rsquo;s local shorthand $c_t:=1/\\sqrt{\\alpha_t}$ and $\\hat b_t:=\\beta_t/\\sqrt{1-\\bar\\alpha_t}$, so $\\tilde\\mu_t=c_t(x_t-\\hat b_t\\epsilon)$ and $\\mu_\\theta=c_t(x_t-\\hat b_t\\epsilon_\\theta)$.\n1. The cancellation. Show $\\|\\tilde\\mu_t-\\mu_\\theta\\|^2=(c_t\\hat b_t)^2\\|\\epsilon-\\epsilon_\\theta\\|^2=\\frac{\\beta_t^2}{\\alpha_t(1-\\bar\\alpha_t)}\\|\\epsilon-\\epsilon_\\theta\\|^2$.\n$\\tilde\\mu_t-\\mu_\\theta=c_t[(x_t-\\hat b_t\\epsilon)-(x_t-\\hat b_t\\epsilon_\\theta)]=c_t\\hat b_t(\\epsilon_\\theta-\\epsilon)$ — the $x_t$ pieces (the only schedule/$x_0$-coupled parts) cancel exactly. Squaring, $\\|\\tilde\\mu_t-\\mu_\\theta\\|^2=(c_t\\hat b_t)^2\\|\\epsilon-\\epsilon_\\theta\\|^2$, and $(c_t\\hat b_t)^2=\\frac{1}{\\alpha_t}\\cdot\\frac{\\beta_t^2}{1-\\bar\\alpha_t}=\\frac{\\beta_t^2}{\\alpha_t(1-\\bar\\alpha_t)}$.\n2. The weight under $\\sigma_t^2=\\tilde\\beta_t$. Dividing by $2\\sigma_t^2$ defines $w_t=\\frac{\\beta_t^2}{2\\sigma_t^2\\alpha_t(1-\\bar\\alpha_t)}$. Simplify $w_t$ when $\\sigma_t^2=\\tilde\\beta_t=\\frac{1-\\bar\\alpha_{t-1}}{1-\\bar\\alpha_t}\\beta_t$.\n$$ w_t=\\frac{\\beta_t^2}{2\\alpha_t(1-\\bar\\alpha_t)\\cdot\\frac{1-\\bar\\alpha_{t-1}}{1-\\bar\\alpha_t}\\beta_t} =\\frac{\\beta_t^2}{2\\alpha_t\\beta_t(1-\\bar\\alpha_{t-1})} =\\frac{\\beta_t}{2\\alpha_t(1-\\bar\\alpha_{t-1})}. $$ 3. Same optimum for any positive weighting. Show that for any weights $\\lambda_t\\gt 0$, the objective $\\sum_t\\lambda_t\\,\\mathbb E\\|\\epsilon-\\epsilon_\\theta(x_t,t)\\|^2$ is minimized by the same predictor $\\epsilon_\\theta(x_t,t)=\\mathbb E[\\epsilon\\mid x_t,t]$.\nEach per-$t$ term $\\mathbb E\\|\\epsilon-\\epsilon_\\theta(x_t,t)\\|^2$ is an MSE, minimized pointwise (for each $x_t$) by the conditional mean $\\mathbb E[\\epsilon\\mid x_t,t]$ — the standard fact that the minimizer of $\\mathbb E\\|Y-g(X)\\|^2$ is $g(X)=\\mathbb E[Y\\mid X]$. A positive weight $\\lambda_t$ scales a term but does not move its minimizer, and (with enough capacity) all per-$t$ minimizers can be achieved simultaneously, so the weighted sum shares the same global minimizer. The weighting changes only finite-capacity dynamics and the implicit objective, not the idealized optimum.\n4. The $v$ inverse map. Verify that $v:=\\sqrt{\\bar\\alpha_t}\\,\\epsilon-\\sqrt{1-\\bar\\alpha_t}\\,x_0$ inverts to $\\epsilon=\\sqrt{1-\\bar\\alpha_t}\\,x_t+\\sqrt{\\bar\\alpha_t}\\,v$, using $x_t=\\sqrt{\\bar\\alpha_t}\\,x_0+\\sqrt{1-\\bar\\alpha_t}\\,\\epsilon$ and $\\bar\\alpha_t+(1-\\bar\\alpha_t)=1$.\nWrite $a_t=\\sqrt{\\bar\\alpha_t},\\,b_t=\\sqrt{1-\\bar\\alpha_t}$ (so $a_t^2+b_t^2=1$). Then\n$$ b_t x_t+a_t v=b_t(a_t x_0+b_t\\epsilon)+a_t(a_t\\epsilon-b_t x_0)=a_t b_t x_0+b_t^2\\epsilon+a_t^2\\epsilon-a_t b_t x_0=(a_t^2+b_t^2)\\epsilon=\\epsilon.\\;\\checkmark $$ Conclusion The $\\epsilon$-parameterization gives $\\mu_\\theta$ the same functional form as $\\tilde\\mu_t$, so the $x_t$ terms cancel exactly and the Gaussian KL collapses to $w_t\\|\\epsilon-\\epsilon_\\theta\\|^2$ with $w_t=\\frac{\\beta_t^2}{2\\sigma_t^2\\alpha_t(1-\\bar\\alpha_t)}$. Replacing $w_t$ by $1$ yields the simplified DDPM loss $L_{\\text{simple}}=\\mathbb E\\,\\|\\epsilon-\\epsilon_\\theta(x_t,t)\\|^2$ — a stable, unit-variance-target regression with an endless supply of training pairs, sitting in a one-parameter-family of SNR weightings (ELBO, Min-SNR-$\\gamma$, P2) that all share the same optimum but differ in finite-capacity dynamics and the likelihood-vs-quality tradeoff. The $\\epsilon$-, $x_0$-, and $v$-targets are exact affine images of one another given $x_t$. In Lesson 6 we open the box labeled $\\epsilon_\\theta$: the U-Net architecture, how the timestep $t$ is injected, and why this network shape suits denoising.\nAll that apparatus folded into three words: guess the noise. The objective is as simple as it will ever be, and infinitely supplied with training pairs. In Lesson 6 we finally lift the lid on the thing doing the guessing — the network $\\epsilon_\\theta$ we have invoked for five lessons without once describing.\nReferences J. Ho, A. Jain, P. Abbeel. Denoising Diffusion Probabilistic Models. NeurIPS 2020. (The $\\epsilon$-parameterization, the cancellation, and $L_{\\text{simple}}$.) D. P. Kingma, T. Salimans, B. Poole, J. Ho. Variational Diffusion Models. NeurIPS 2021. (Continuous-time ELBO as SNR-weighted denoising; the role of $\\mathrm{SNR}(t)$.) T. Salimans, J. Ho. Progressive Distillation for Fast Sampling of Diffusion Models. ICLR 2022. (The $v$-parameterization and its rotation geometry.) T. Hang, S. Gu, C. Li, et al. Efficient Diffusion Training via Min-SNR Weighting Strategy. ICCV 2023. (Min-SNR-$\\gamma$ reweighting.) J. Choi, J. Lee, C. Shin, et al. Perception Prioritized Training of Diffusion Models (P2). CVPR 2022. (P2 SNR-based weighting and the content-bearing noise levels.) ← Previous: Lesson 4 — The Variational Lower Bound · Course Index · Next: Lesson 6 — The U-Net Architecture → · Appendix A: Acronyms · Appendix B: Formulas The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/tutorials/diffusion-lessons/lesson-05-simplified-objective/","summary":"\u003ch1 id=\"lesson-5--the-simplified-training-objective\"\u003eLesson 5 — The Simplified Training Objective\u003c/h1\u003e\n\u003cblockquote\u003e\n\u003cp\u003e\u003cstrong\u003eRecap of Lesson 4.\u003c/strong\u003e We maximize a variational lower bound $L = L_T + \\sum_{t\\ge2}L_{t-1} + L_0$. The prior term $L_T$ is a parameter-free constant; the reconstruction term $L_0$ is minor; the \u003cstrong\u003edenoising terms\u003c/strong\u003e $L_{t-1}$ carry the learning. With fixed reverse variance $\\sigma_t^2 I$, each reduces to a weighted squared error between the network\u0026rsquo;s reverse mean $\\mu_\\theta(x_t,t)$ and the analytic posterior mean $\\tilde\\mu_t(x_t,x_0)$: $L_{t-1}=\\frac{1}{2\\sigma_t^2}\\|\\tilde\\mu_t-\\mu_\\theta\\|^2 + C$. Now we collapse this into one clean loss.\u003c/p\u003e","title":"The Simplified Objective"},{"content":"Lesson 6 — Expert Capacity, Token Dropping \u0026amp; the Training Forward Pass Technical course on MoE LLM mechanics — Lesson 6 of 10.\nRecap of Lesson 5 A sparse router collapses without help, so MoE training adds an explicit balancing pressure: the Switch auxiliary loss $\\alpha E\\sum_i f_iP_i$ (load $\\times$ mass), the router z-loss for logit stability, or aux-loss-free bias control. These shape the router\u0026rsquo;s intent toward uniform usage. But intent is not enough: the dispatched tokens must physically fit into expert buffers on real hardware, and those buffers have a fixed size chosen in advance. This lesson introduces expert capacity, the token dropping that happens on overflow, and the batched dispatch/combine tensors that turn the per-token math of Lesson 4 into an efficient parallel operation.\nLessons 4–5 treated routing one token at a time. Real training processes a whole batch at once on parallel hardware, and hardware wants fixed-shape, equal-size tensors — but routing produces variable, data-dependent expert loads. Reconciling \u0026ldquo;the router sends whatever it wants\u0026rdquo; with \u0026ldquo;each expert has a fixed-size slot\u0026rdquo; is what this lesson is about, and it forces a uniquely MoE decision: what to do when an expert is oversubscribed.\n1. Why a fixed capacity is necessary On a GPU/TPU, the experts are run as a single batched matrix multiply: you gather all tokens assigned to expert $i$ into a contiguous buffer and multiply by $W_1^{(i)},W_2^{(i)}$. For this to compile to a static, efficient kernel — and, in distributed training, for the all-to-all communication of Lesson 7 to have a fixed message size — every expert\u0026rsquo;s buffer must have the same, fixed number of slots, decided before the batch runs. That fixed slot count is the expert capacity $C$.\nBut routing is data-dependent: in any given batch, some experts will be assigned more tokens than others (perfect balance is the goal, never the reality). So two things can go wrong against a fixed $C$: an expert receives more than $C$ tokens (overflow — some must be dropped) or fewer than $C$ (underflow — the buffer is padded with zeros and the spare compute is wasted). Capacity is the dial that trades these two off.\n2. The capacity factor With $T$ tokens in a batch, $E$ experts, and top-$k$ routing, the average number of tokens per expert is $kT/E$ (each token goes to $k$ experts). The capacity is this average scaled by a capacity factor $C_f\\ge 1$ ([CONFIG]):\n$$ \\boxed{\\ C=\\left\\lceil C_f\\cdot\\frac{k\\,T}{E}\\right\\rceil\\ } $$ $C_f=1.0$ means buffers exactly fit the average load — any imbalance causes dropping. $C_f=1.25$ (a common training value) gives each expert 25% headroom, tolerating moderate imbalance at the price of 25% wasted expert compute when balanced. Larger $C_f$ drops fewer tokens but wastes more compute and memory on padding. $C_f$ is the explicit knob for the drop-vs-waste trade-off. Total expert compute scales with $E\\cdot C=C_f kT$, so raising $C_f$ raises cost linearly — sparsity\u0026rsquo;s FLOP savings are partly given back to slack capacity. Choosing $C_f$ is a uniquely MoE tuning decision with no dense counterpart.\nFigure 1. The capacity factor is a drop-versus-waste dial.\n3. Token dropping When expert $i$ is assigned more than $C$ tokens in a batch, the surplus tokens are dropped from that expert: typically the lowest-gate-weight assignments past the $C$-th are not processed by expert $i$. Concretely, with the residual connection in place, a dropped token\u0026rsquo;s MoE sub-layer output is just the residual itself — the token \u0026ldquo;skips\u0026rdquo; the expert computation and passes through unchanged:\n$$ H'_t=\\tilde H_t+\\sum_{i\\in\\mathcal T_k(\\mathbf x_t)\\ \\cap\\ \\{\\text{not overflowed}\\}} g_i(\\mathbf x_t)\\,E_i(\\mathbf x_t), $$so a fully-dropped token contributes nothing from this layer and relies on the residual stream and the other layers. Dropping is not a bug to be eliminated at all costs — a well-balanced router with reasonable $C_f$ drops few tokens, and the residual path makes the occasional drop survivable. But heavy dropping (a sign of poor balance, hence the link back to Lesson 5\u0026rsquo;s losses) degrades quality, which is why drop rate is a key training diagnostic. Some modern recipes use dropless MoE (variable-size grouped kernels, e.g. MegaBlocks) that avoid fixed capacity entirely at the cost of more complex kernels.\nContrast with the dense model. A dense FFN processes every token — there is no capacity, no overflow, and certainly no token that silently skips the sub-layer. The very concepts of \u0026ldquo;expert capacity,\u0026rdquo; \u0026ldquo;capacity factor,\u0026rdquo; and \u0026ldquo;dropped tokens\u0026rdquo; exist only because routing produces variable per-expert loads that must be squeezed into fixed-shape hardware buffers. A dense model\u0026rsquo;s per-layer compute is exactly determined by its shape; an MoE\u0026rsquo;s is determined by its shape and the runtime routing pattern, which is why it needs slack and a drop policy.\nFigure 2. What overflow looks like, and what balance has to do with it.\n4. The batched forward pass: dispatch and combine tensors To run all experts as parallel batched matmuls, the per-token routing is expressed as two sparse tensors ([STATE], built fresh each batch):\nThe dispatch tensor $D\\in\\{0,1\\}^{T\\times E\\times C}$ scatters tokens into expert buffers: $D_{t,i,c}=1$ if token $t$ is the $c$-th token assigned to expert $i$. Multiplying the input by $D$ produces the per-expert input buffers $X_{\\text{exp}}\\in\\mathbb R^{E\\times C\\times d}$ — fixed shape, ready for batched matmul. The combine tensor $\\tilde D\\in\\mathbb R^{T\\times E\\times C}$ is the same scatter pattern but carrying the gate weights $g_i$ in the nonzero positions, so multiplying the expert outputs by $\\tilde D$ both un-scatters them back to token order and applies the convex weighting of Lesson 4 in one operation. The forward pass is then three dense operations: $X_{\\text{exp}}=D^\\top X$ (dispatch), $Y_{\\text{exp}}=\\mathrm{Experts}(X_{\\text{exp}})$ (parallel expert FFNs), $Y=\\tilde D\\,Y_{\\text{exp}}$ (combine). This formulation is what makes sparse routing run efficiently on dense-matmul hardware, and — as the next lesson shows — the dispatch and combine steps are exactly where the cross-device communication happens when experts live on different accelerators.\nflowchart LR X[\"tokens X(T × d)\"] --\u003e|\"dispatch Dᵀ\"| XE[\"expert buffers(E × C × d)\"] XE --\u003e|\"parallel FFNs\"| YE[\"expert outputs(E × C × d)\"] YE --\u003e|\"combine D̃ (gate-weighted)\"| Y[\"layer output(T × d)\"] Figure 3. Sparse routing, executed as three dense matmuls.\n5. Learned / config / state Quantity Category Expert and router weights [LEARNED] Capacity factor $C_f$, derived capacity $C$, drop policy, dropless vs. fixed-capacity [CONFIG] Dispatch tensor $D$, combine tensor $\\tilde D$, per-expert buffers, drop mask, drop rate [STATE] Conclusion Real hardware wants fixed-shape buffers, but routing produces variable per-expert loads, so each expert is given a fixed capacity $C=\\lceil C_f\\,kT/E\\rceil$ set by the capacity factor $C_f$ — the explicit drop-vs-waste dial ($C_f=1$ risks dropping, large $C_f$ wastes padding compute). Tokens beyond an expert\u0026rsquo;s capacity are dropped, falling back on the residual stream; the drop rate is a core diagnostic tied directly to router balance (Lesson 5). The batch is executed via a dispatch tensor (scatter tokens into fixed buffers) and a combine tensor (gate-weighted un-scatter), turning per-token routing into three dense matmuls. None of this — capacity, dropping, dispatch/combine — exists in a dense model, whose per-layer compute is fully fixed by its shape.\nThe dispatch and combine steps assumed all experts sit on one device. At real scale they do not: the experts are sharded across many accelerators, and dispatch/combine become network operations. The next lesson is about that communication — the distinctive bottleneck of MoE training.\n← Lesson 5 — Load Balancing \u0026amp; Auxiliary Losses · Course index · Next: Lesson 7 — Distributed Training: Expert Parallelism → Appendices: A — MoE Family · B — Dense vs. MoE · C — Acronyms The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/tutorials/moe-lessons/lesson-06-capacity-and-token-dropping/","summary":"\u003ch1 id=\"lesson-6--expert-capacity-token-dropping--the-training-forward-pass\"\u003eLesson 6 — Expert Capacity, Token Dropping \u0026amp; the Training Forward Pass\u003c/h1\u003e\n\u003cp\u003e\u003cem\u003eTechnical course on MoE LLM mechanics — Lesson 6 of 10.\u003c/em\u003e\u003c/p\u003e\n\u003chr\u003e\n\u003ch2 id=\"recap-of-lesson-5\"\u003eRecap of Lesson 5\u003c/h2\u003e\n\u003cp\u003eA sparse router collapses without help, so MoE training adds an explicit balancing pressure: the Switch auxiliary loss $\\alpha E\\sum_i f_iP_i$ (load $\\times$ mass), the router z-loss for logit stability, or aux-loss-free bias control. These shape the router\u0026rsquo;s \u003cem\u003eintent\u003c/em\u003e toward uniform usage. But intent is not enough: the dispatched tokens must physically fit into expert buffers on real hardware, and those buffers have a fixed size chosen in advance. This lesson introduces \u003cstrong\u003eexpert capacity\u003c/strong\u003e, the \u003cstrong\u003etoken dropping\u003c/strong\u003e that happens on overflow, and the batched \u003cstrong\u003edispatch/combine\u003c/strong\u003e tensors that turn the per-token math of Lesson 4 into an efficient parallel operation.\u003c/p\u003e","title":"Expert Capacity, Token Dropping and the Training Forward Pass"},{"content":"Lesson 6 — From Representations to Predictions Technical course on encoder LLM mechanics — Lesson 6 of 10.\nRecap of Lesson 5 An encoder is pretrained by masked language modeling: corrupt ~15% of tokens (the 80/10/10 recipe), run the bidirectional stack, and reconstruct the originals with a cross-entropy loss over masked positions only. NSP was an optional second objective, later dropped. But Lesson 5 referred to an \u0026ldquo;MLM head\u0026rdquo; that turns a top-layer vector into a vocabulary distribution, and Lesson 4 left [CLS] as a summary register with nothing attached. This lesson builds the heads: the small networks that convert the encoder\u0026rsquo;s $H^{(N)}\\in\\mathbb R^{n\\times d}$ into actual predictions — and shows how a decoder\u0026rsquo;s single head becomes an encoder\u0026rsquo;s family of heads.\n1. The setup: representations are not predictions Recall the encoder\u0026rsquo;s output is $H^{(N)}\\in\\mathbb R^{n\\times d}$ — context-aware vectors, no probabilities. A head is a (usually tiny) function mapping those vectors to a task\u0026rsquo;s output space. The decoder has essentially one head, used everywhere: the unembedding to vocabulary logits. The encoder has different heads for different shapes of task, and choosing/attaching the right one is most of what \u0026ldquo;using an encoder\u0026rdquo; means. We meet four: the MLM head (used in pretraining), and the three fine-tuning heads — sequence-level, token-level, and span.\n2. The MLM head, and weight tying During pretraining (Lesson 5), each masked position\u0026rsquo;s top vector $H^{(N)}_{i,:}$ must become a distribution over the $V$-token vocabulary. The MLM head is a small transform followed by a projection to vocabulary logits:\n$$ \\hat{\\mathbf p}_i=\\mathrm{softmax}\\big(W_U\\,\\underbrace{\\big[\\mathrm{LayerNorm}(\\mathrm{GELU}(W_d\\,H^{(N)}_{i,:}+\\mathbf b_d))\\big]}_{\\phi(\\cdot)\\in\\mathbb R^{d}}+\\mathbf b_U\\big), $$where $W_d\\in\\mathbb R^{d\\times d}$ is a small \u0026ldquo;transform\u0026rdquo; layer and $W_U\\in\\mathbb R^{V\\times d}$ is the projection to logits. This $W_U$ is the encoder\u0026rsquo;s unembedding, and it is structurally identical to the decoder\u0026rsquo;s: a $V\\times d$ matrix mapping a hidden vector to one logit per vocabulary token.\nWeight tying. Just as in the decoder, $W_U$ is typically tied to the input token-embedding matrix $E_{\\text{tok}}$ (Lesson 2): $W_U=E_{\\text{tok}}$. This saves $Vd\\approx23$M parameters in BERT-base and reflects a clean symmetry — the same matrix maps token→vector (input) and vector→token-logit (output). The gradient $\\partial\\mathcal L/\\partial\\mathbf z=\\hat{\\mathbf p}-\\mathbf y$ identity from the decoder course applies unchanged at each masked position.\nContrast with the decoder. The MLM head and the decoder\u0026rsquo;s LM head are nearly the same object — a (tied) projection to vocabulary logits. The difference is where and when it is applied: the decoder applies it at every position at both train and inference time (it is the model\u0026rsquo;s output); the encoder applies it only at masked positions, only during pretraining, and then usually throws it away — fine-tuning replaces it with a task head. The MLM head is scaffolding for learning representations, not the product.\nFigure 1. The MLM head, and the matrix it borrows.\n3. Pooling: turning $n$ vectors into one Most encoder tasks are sequence-level (one label per input: sentiment, entailment, a retrieval embedding). They need a single vector summarizing the whole sequence, so we must pool the $n$ token vectors into one. Two standard choices:\n[CLS] pooling. Take the top-layer vector at position 0: $\\mathbf c = H^{(N)}_{0,:}$. Often passed through a learned \u0026ldquo;pooler\u0026rdquo; — a dense layer with $\\tanh$: $\\mathbf c'=\\tanh(W_p\\mathbf c+\\mathbf b_p)$. This is BERT\u0026rsquo;s default. It works because, as Lesson 4 argued, the bidirectional [CLS] position can attend to everything and is trained (via NSP, then fine-tuning) to aggregate.\nMean pooling. Average the token vectors over real (non-padding) positions:\n$$ \\mathbf m=\\frac{1}{|\\mathcal R|}\\sum_{i\\in\\mathcal R}H^{(N)}_{i,:},\\qquad \\mathcal R=\\text{real-token positions}. $$Mean pooling often produces better sentence embeddings than raw [CLS], especially for similarity/retrieval, because it uses every token\u0026rsquo;s information rather than relying on one register. Sentence-BERT (Lesson 10) found mean pooling superior for semantic-similarity embeddings. The choice is a [CONFIG] decision tied to the downstream task.\nContrast with the decoder. A decoder needs no pooling step: causal masking makes the last token\u0026rsquo;s vector the natural sequence summary (it alone has attended to the entire input), so \u0026ldquo;sequence representation = last hidden state\u0026rdquo; is automatic. An encoder, being bidirectional, has many candidate summary positions and must choose a pooling rule — [CLS], mean, or max — as an explicit design decision.\nFigure 2. Two ways to pool n vectors into one.\n4. The three fine-tuning head shapes Lesson 9 covers training these; here we fix their shapes, because the shape is what makes the encoder a general \u0026ldquo;understanding\u0026rdquo; engine.\nSequence-level head (one label per input). Pool to a single vector, then a linear classifier:\n$$ \\hat{\\mathbf y}=\\mathrm{softmax}(W_{\\text{cls}}\\,\\mathbf c'+\\mathbf b),\\qquad W_{\\text{cls}}\\in\\mathbb R^{C\\times d}, $$for $C$ classes. Used for sentiment, topic, intent, entailment (sentence-pair, via the [SEP]-joined input of Lesson 2). For regression (e.g. similarity score), drop the softmax and output a scalar.\nToken-level head (one label per token). Apply the same classifier to every token vector independently:\n$$ \\hat{\\mathbf y}_i=\\mathrm{softmax}(W_{\\text{tok}}\\,H^{(N)}_{i,:}+\\mathbf b),\\qquad i=1,\\dots,n. $$Used for named-entity recognition, part-of-speech tagging, and any sequence-labeling task. This head exploits the encoder\u0026rsquo;s defining property directly: every output vector is a fully contextualized description of its token, so a per-token linear layer suffices.\nSpan head (two positions). For extractive QA — given a question and a passage (joined with [SEP]), find the answer span in the passage. Learn two vectors $\\mathbf s,\\mathbf e\\in\\mathbb R^{d}$ (start and end), and score each position:\n$$ P_{\\text{start}}(i)=\\frac{\\exp(\\mathbf s^\\top H^{(N)}_{i,:})}{\\sum_j\\exp(\\mathbf s^\\top H^{(N)}_{j,:})},\\qquad P_{\\text{end}}(i)=\\frac{\\exp(\\mathbf e^\\top H^{(N)}_{i,:})}{\\sum_j\\exp(\\mathbf e^\\top H^{(N)}_{j,:})}. $$The predicted answer is the span $(i^\\star,j^\\star)$ with $i^\\star\\le j^\\star$ maximizing $P_{\\text{start}}(i)P_{\\text{end}}(j)$. This is extractive — it points at text already present — which is the encoder\u0026rsquo;s natural mode.\nflowchart TD H[\"H^(N): n × d contextual vectors\"] --\u003e P1[\"pool → c\"] P1 --\u003e S[\"sequence head:1 label (sentiment, NLI)\"] H --\u003e T[\"per-token head:n labels (NER, POS)\"] H --\u003e SP[\"span head:start/end positions (extractive QA)\"] Contrast with the decoder. A decoder does all of these tasks through one head — the vocabulary projection — by casting every task as text generation (\u0026ldquo;the sentiment is: positive\u0026rdquo;). The encoder instead swaps in a small task-specific head matched to the output\u0026rsquo;s shape (a label, a tag sequence, two indices). Neither is strictly better: the decoder\u0026rsquo;s single-head approach is wonderfully general and needs no architecture change per task; the encoder\u0026rsquo;s typed heads are smaller, faster, calibrated, and produce structured outputs (a probability over $C$ classes, not a sampled string) that are easier to threshold and evaluate.\nFigure 3. Three head shapes, and the regression variant.\n5. Why typed outputs matter The encoder\u0026rsquo;s heads produce structured, bounded outputs: a categorical distribution over exactly $C$ classes, a tag per token, or two valid positions. That has practical consequences a generative decoder cannot match cheaply. A classification probability is directly usable as a confidence and is trivially calibrated/thresholded; a span is guaranteed to be a substring of the input (no hallucination possible); NER tags are constrained to the label set. A decoder asked to \u0026ldquo;output the sentiment\u0026rdquo; can in principle emit anything, must be parsed, can hallucinate a label outside the set, and gives a probability over token strings rather than over the task\u0026rsquo;s classes. For high-volume, structured prediction — content moderation, search ranking, entity tagging over billions of documents — the encoder\u0026rsquo;s typed, single-pass, non-generative output is both cheaper and safer.\nConclusion The encoder\u0026rsquo;s output $H^{(N)}$ becomes predictions through heads. Pretraining uses the MLM head — a small transform plus a vocabulary projection $W_U$, usually tied to the input embedding — applied only at masked positions, then discarded. Sequence-level tasks first pool the $n$ vectors into one ([CLS] or mean pooling — a choice the decoder never faces, since its last token is the automatic summary), then apply a small classifier. Token-level tasks apply a shared classifier to every contextual vector; span tasks score start/end positions. Where the decoder routes every task through one vocabulary-projection head by generating text, the encoder swaps in a typed head matched to each task\u0026rsquo;s output shape — yielding smaller, faster, calibrated, structured predictions at the cost of needing a new head (and fine-tuning) per task.\nWe can now both train representations (Lesson 5) and read predictions out of them (this lesson). Lesson 7 asks the measurement question: with no generation and no standard perplexity, how do you tell whether an encoder is any good?\n← Previous: Lesson 5 — Masked Language Modeling · Course index · Next: Lesson 7 — Evaluating an Encoder → Appendices: A — Encoder Family · B — Encoder vs. Decoder Cheat Sheet · C — Acronyms The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/tutorials/encoder-deep-dive/lesson-06-representations-to-predictions/","summary":"\u003ch1 id=\"lesson-6--from-representations-to-predictions\"\u003eLesson 6 — From Representations to Predictions\u003c/h1\u003e\n\u003cp\u003e\u003cem\u003eTechnical course on encoder LLM mechanics — Lesson 6 of 10.\u003c/em\u003e\u003c/p\u003e\n\u003chr\u003e\n\u003ch2 id=\"recap-of-lesson-5\"\u003eRecap of Lesson 5\u003c/h2\u003e\n\u003cp\u003eAn encoder is pretrained by \u003cstrong\u003emasked language modeling\u003c/strong\u003e: corrupt ~15% of tokens (the 80/10/10 recipe), run the bidirectional stack, and reconstruct the originals with a cross-entropy loss over masked positions only. NSP was an optional second objective, later dropped. But Lesson 5 referred to an \u0026ldquo;MLM head\u0026rdquo; that turns a top-layer vector into a vocabulary distribution, and Lesson 4 left \u003ccode\u003e[CLS]\u003c/code\u003e as a summary register with nothing attached. This lesson builds the \u003cstrong\u003eheads\u003c/strong\u003e: the small networks that convert the encoder\u0026rsquo;s $H^{(N)}\\in\\mathbb R^{n\\times d}$ into actual predictions — and shows how a decoder\u0026rsquo;s \u003cem\u003esingle\u003c/em\u003e head becomes an encoder\u0026rsquo;s \u003cem\u003efamily\u003c/em\u003e of heads.\u003c/p\u003e","title":"From Representations to Predictions"},{"content":"Lesson 6 — Prefix-LM: Encoder–Decoder as a Special Case Technical course on encoder–decoder transformers — Lesson 6 of 10.\nRecap of Lesson 5 The full model is a stack of two-sublayer encoder blocks and three-sublayer decoder blocks; the encoder runs once into $H_{\\text{enc}}$, broadcast to every decoder block\u0026rsquo;s cross-attention. Parameters scale as $12N_{\\text{enc}}d^2+16N_{\\text{dec}}d^2+Vd$. T5 and BART differ from the textbook only in [CONFIG] dressings (positions, norm, activation, objective), never in the cross-attention core. We now zoom out: the two-stack design is one point on a spectrum, and a single mask connects it to the decoder-only model.\n1. The question: do we need two stacks? Lesson 1 argued why two stacks help — bidirectional encoding and a compute-once memory. But decoder-only models clearly can do translation and summarization by concatenation. So the honest question is: is the encoder–decoder a fundamentally different object, or a special arrangement of the same one? The answer is the latter, and the bridge is a masking trick called the prefix language model (prefix-LM). Understanding it dissolves the apparent gulf between the two families and explains why the field could later drop encoder–decoders for most tasks without losing capability.\n2. The unifying view: it\u0026rsquo;s all about the mask Recall that the only thing distinguishing encoder self-attention from decoder self-attention was the mask $M$: $M=0$ (bidirectional) for the encoder, $M=M^{\\text{causal}}$ (lower-triangular) for the decoder. Now put input and output into one sequence $[\\,x\\,;\\,y\\,]$ of length $n+m$ and run a single stack over it, but with a block-structured mask that gives the input part bidirectional attention and the output part causal attention:\n$$ M^{\\text{prefix}}_{ij}= \\begin{cases} 0, \u0026 j\\le n \\quad\\text{(any token may attend to the whole input prefix)}\\\\[2pt] 0, \u0026 n\\lt j\\le i \\quad\\text{(output token attends to earlier output)}\\\\[2pt] -\\infty, \u0026 j\\gt i \\text{ and } j\\gt n \\quad\\text{(no peeking at future output)} \\end{cases} $$In words: the first $n$ positions (the prefix $x$) are mutually fully visible — like an encoder; positions $n+1\\dots n+m$ (the continuation $y$) see the entire prefix and all earlier output but not future output — like a decoder. This is the prefix-LM mask, and it gives a single-stack model the two behaviors the two-stack model split apart.\nflowchart LR subgraph M[\"Prefix-LM mask over [x ; y]\"] direction TB pp[\"x↔x block: bidirectional (0)\"] yp[\"y→x block: full visibility (0)\"] yy[\"y↔y block: causal (lower-tri)\"] fy[\"y→future-y: masked (−∞)\"] end A model trained this way conditions its generation of $y$ on a bidirectionally-encoded $x$ — the key advantage of the encoder — without a separate stack and without cross-attention. The input is encoded by the same self-attention that generates, just under a more permissive mask for the prefix region.\nFigure 1. One mask family: bidirectional, prefix, causal.\n3. How prefix-LM relates to the two-stack model The two designs compute closely related functions, and it is worth being precise about what is shared and what differs.\nWhat is shared. Both give $x$ bidirectional context and generate $y$ causally conditioned on all of $x$. Both factorize $P_\\theta(y\\mid x)=\\prod_t P_\\theta(y_t\\mid y_{\\lt t},x)$.\nWhat differs.\nParameters. The two-stack model has separate weights for reading and writing, plus dedicated cross-attention projections. The prefix-LM reuses one set of weights for both roles and conditions through ordinary self-attention — no $W^c_{Q,K,V}$ at all. Conditioning is \u0026ldquo;free\u0026rdquo; in parameter terms but shares capacity between reading and writing. Compute reuse. The two-stack encoder produces $H_{\\text{enc}}$ once; in a prefix-LM the prefix\u0026rsquo;s keys/values are likewise computed once and cached, so per-step decoding cost is similar. But the prefix-LM recomputes nothing extra — there is no second stack to run. Depth of conditioning. In the two-stack model, conditioning is injected at every decoder layer via cross-attention; in the prefix-LM it is mediated by the shared representation. Empirically the two-stack design tends to be a bit stronger on hard seq2seq tasks (translation, summarization), which is part of why T5 chose it; prefix-LM is competitive and simpler. So the encoder–decoder is the prefix-LM with the prefix and continuation given their own parameter sets and bonded by cross-attention. Conversely, a prefix-LM is an encoder–decoder with the seam removed and weights tied. They are two points on one design axis.\nFigure 2. Two stacks or one, and what changes.\n4. The full spectrum: one mask family, three architectures Step back one more notch and the entire landscape of Lesson 1 collapses into a single statement: the three transformer families differ only in their attention mask (and whether $K,V$ come from a second sequence). Everything else is shared.\nArchitecture Mask over the relevant region Conditioning Canonical models Encoder-only fully bidirectional ($M=0$) none (it only reads) BERT, RoBERTa Encoder–decoder bidirectional enc + causal dec + cross-attn explicit, per decoder layer T5, BART, original Transformer Prefix-LM bidirectional prefix + causal suffix (one stack) implicit, shared weights UniLM, T5\u0026rsquo;s \u0026ldquo;prefix-LM\u0026rdquo; baseline, GLM Decoder-only fully causal implicit, via concatenated prefix GPT, Llama, Mistral Read top to bottom, the mask grows steadily more causal and the conditioning steadily more implicit. The encoder-only model is \u0026ldquo;all bidirectional, no generation\u0026rdquo;; the decoder-only is \u0026ldquo;all causal, generation by continuation\u0026rdquo;; the encoder–decoder and prefix-LM sit in between, mixing the two regimes — the former by separating the regimes into two stacks, the latter by splitting one sequence with a block mask.\nThis is the deep reason a modern decoder-only LLM can translate and summarize despite having no encoder: the causal mask plus a well-chosen prompt approximates the prefix-LM\u0026rsquo;s behavior, and the prefix-LM approximates the encoder–decoder\u0026rsquo;s. What the decoder-only gives up — bidirectional input encoding and a clean compute-once memory — turned out to matter less than the simplicity and scaling convenience of a single uniform stack, which is why the field largely consolidated on decoder-only for general-purpose LLMs while keeping encoder–decoders where their inductive bias pays off (Lesson 10).\nFigure 3. The spectrum, read from all-bidirectional to all-causal.\n5. T5\u0026rsquo;s experiment, briefly The T5 paper ran exactly this comparison as a controlled experiment — same data, same compute, varying only the architecture/mask — and found the encoder–decoder with span-corruption pretraining the strongest configuration for its transfer-learning benchmark, with the prefix-LM and decoder-only variants close but behind on the seq2seq-heavy tasks. That result is the empirical anchor under this whole lesson: the spectrum is real, the points are genuinely interchangeable in principle, and the two-stack design earns its extra cross-attention parameters on tasks where reading the input well is decisive.\nLearned / config / state Quantity Category Shared stack weights (prefix-LM) or separate enc/dec weights (two-stack) [LEARNED] Choice of mask family (bidirectional / prefix / causal); one-stack vs. two-stack [CONFIG] The block-structured mask $M^{\\text{prefix}}$ (data/length dependent) [CONFIG]/[STATE] Hidden states, attention matrices [STATE] Conclusion The encoder–decoder is not a different species — it is one point on a masking spectrum. Put $[\\,x\\,;\\,y\\,]$ in a single stack under a prefix-LM mask (bidirectional over the prefix $x$, causal over the continuation $y$) and you condition generation on a bidirectionally-encoded input without a second stack or cross-attention. The encoder–decoder is the prefix-LM with reading and writing given separate parameters and bonded by cross-attention; the decoder-only is the same idea with a fully causal mask and implicit, prompt-based conditioning. All three families — and encoder-only at the far end — differ only in their attention mask and where $K,V$ come from. T5\u0026rsquo;s own ablation found the two-stack design strongest for seq2seq transfer, which is the empirical reason the architecture earns its keep.\nThat closes the architectural arc. The remaining lessons are about use: how this object is trained (Lessons 7–8) and how it generates (Lesson 9). We begin with the objective that made T5 and BART work — denoising pretraining — which is where their stories most sharply diverge from both encoder-only MLM and decoder-only next-token prediction.\n← Lesson 5 — The Encoder–Decoder Block · Course index · Next: Lesson 7 — Pretraining Objectives → Appendices: A — Formulas · B — Architecture Comparison · C — Acronyms The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/tutorials/encoder-decoder-lessons/lesson-06-prefix-lm/","summary":"\u003ch1 id=\"lesson-6--prefix-lm-encoderdecoder-as-a-special-case\"\u003eLesson 6 — Prefix-LM: Encoder–Decoder as a Special Case\u003c/h1\u003e\n\u003cp\u003e\u003cem\u003eTechnical course on encoder–decoder transformers — Lesson 6 of 10.\u003c/em\u003e\u003c/p\u003e\n\u003chr\u003e\n\u003ch2 id=\"recap-of-lesson-5\"\u003eRecap of Lesson 5\u003c/h2\u003e\n\u003cp\u003eThe full model is a stack of two-sublayer encoder blocks and three-sublayer decoder blocks; the encoder runs once into $H_{\\text{enc}}$, broadcast to every decoder block\u0026rsquo;s cross-attention. Parameters scale as $12N_{\\text{enc}}d^2+16N_{\\text{dec}}d^2+Vd$. T5 and BART differ from the textbook only in \u003cstrong\u003e[CONFIG]\u003c/strong\u003e dressings (positions, norm, activation, objective), never in the cross-attention core. We now zoom out: the two-stack design is one point on a spectrum, and a single mask connects it to the decoder-only model.\u003c/p\u003e","title":"Prefix-LM: Encoder–Decoder as a Special Case"},{"content":"Lesson 6 — The Loss Function Technical course on LLM mechanics — Lesson 6 of 10.\nRecap of Lesson 5 The head produces logits $\\mathbf z=\\mathbf h^\\top W_U$ and softmax gives $\\hat{\\mathbf p}\\in\\Delta^{V-1}$. The objective is maximum likelihood of observed tokens = minimum average cross-entropy $\\mathcal L=-\\frac1n\\sum_t\\log\\hat p_{t,x_t}$, self-supervised because the label at $t$ is $x_{t+1}$. The softmax+cross-entropy pairing yields the clean gradient $\\partial\\mathcal L/\\partial\\mathbf z=\\hat{\\mathbf p}-\\mathbf y$. This lesson develops the loss information-theoretically, derives its key properties, and connects it to perplexity.\nLesson 5 introduced cross-entropy as the thing to minimize and largely took it on faith. Here we earn it. Understanding what the loss measures — and what it can never do better than — tells us what training is actually accomplishing, why this loss rather than accuracy, and how to read the number off a training curve. It is the last conceptual stop before we start turning the crank on $\\theta$.\n1. Cross-entropy, entropy, KL For a true distribution $p$ and model distribution $\\hat p$ over the vocabulary, cross-entropy is\n$$ H(p,\\hat p)=-\\sum_{w=1}^{V}p_w\\log\\hat p_w. $$It decomposes exactly into the entropy of $p$ plus the Kullback–Leibler divergence:\n$$ H(p,\\hat p)=\\underbrace{-\\sum_w p_w\\log p_w}_{H(p)}\\;+\\;\\underbrace{\\sum_w p_w\\log\\frac{p_w}{\\hat p_w}}_{D_{\\mathrm{KL}}(p\\,\\|\\,\\hat p)}. $$This is the cleanest way to see what training does. $H(p)$ is the irreducible uncertainty of language itself — it does not depend on $\\theta$. $D_{\\mathrm{KL}}(p\\|\\hat p)\\ge0$ (Gibbs\u0026rsquo; inequality), with equality iff $\\hat p=p$. Therefore\n$$ \\min_\\theta H(p,\\hat p_\\theta)\\iff \\min_\\theta D_{\\mathrm{KL}}(p\\,\\|\\,\\hat p_\\theta), $$i.e. minimizing cross-entropy is minimizing the KL divergence from the model to the true language distribution. Training can never push the loss below $H(p)$ — the entropy of language is the noise floor.\nThe one-hot collapse In practice we don\u0026rsquo;t know $p$; we have samples. The empirical target at a position is the one-hot $\\mathbf y$ (the token that actually occurred), so $p_w=\\delta_{w,x_t}$ and the sum collapses:\n$$ H(\\mathbf y,\\hat{\\mathbf p})=-\\sum_w \\delta_{w,x_t}\\log\\hat p_w=-\\log\\hat p_{x_t}. $$Averaged over the corpus this is exactly the MLE loss of Lesson 5. (With label smoothing or distillation the target is not one-hot, and all $V$ terms survive — the general form matters there.)\nThe decomposition tells us what training targets in principle; the collapsed one-hot form is what we actually compute per token. It is worth looking closely at how that single term behaves as the model\u0026rsquo;s confidence varies, because its shape is the reason cross-entropy is the right thing to descend at all.\nFigure 1. The loss floor you cannot train through.\n2. Properties of the per-token loss $\\ell=-\\log\\hat p_{x_t}$ Confidence on true token $\\hat p_{x_t}$ Loss $\\ell$ (nats) Behavior $1.0$ $0$ perfect, confident $0.5$ $0.693$ uncertain $0.1$ $2.303$ wrong-ish $\\to 0$ $\\to\\infty$ confidently wrong (unbounded penalty) The unbounded penalty as $\\hat p_{x_t}\\to0$ is the proper-scoring-rule property: the loss is minimized in expectation only by reporting calibrated probabilities, not just by ranking the right token first. A model that says \u0026ldquo;99%\u0026rdquo; must be right ~99% of the time or pay heavily. This is why cross-entropy, not accuracy, is the training signal — accuracy is flat almost everywhere (zero gradient), while cross-entropy provides a smooth, informative gradient at every prediction.\nFigure 2. Why the training signal is cross-entropy and not accuracy.\n3. Units: nats vs. bits The loss is a logarithm, so its numerical value depends on the base of that logarithm — and the choice of base is more than bookkeeping, because one of them carries a concrete interpretation.\nWith natural log the unit is nats; with $\\log_2$ it is bits ($1\\text{ nat}=1/\\ln 2\\approx1.443$ bits). Bits have a coding interpretation: a loss of $b$ bits/token means the model could compress that text to $b$ bits per token under an optimal arithmetic code. This is the literal link between language modeling and compression — a better LLM is a better text compressor, and benchmarks like bits-per-byte exploit exactly this.\n4. The corpus loss and its gradient So far the loss has been per token. Optimization actually proceeds over batches, so we need the aggregate quantity the optimizer will see — and the way it aggregates has a quiet but practical consequence.\n$$ \\mathcal L(\\theta)=-\\frac{1}{|B|}\\sum_{(x_{\\lt t},x_t)\\in B}\\log P_\\theta(x_t\\mid x_{\\lt t}), $$averaged over a batch $B$. The averaging (rather than summing) decouples the gradient scale from batch size, so a chosen learning rate transfers across batch sizes. From Lesson 5, the gradient at each position\u0026rsquo;s logits is $\\hat{\\mathbf p}-\\mathbf y$; backprop (Lesson 7) carries that through the network to $\\nabla_\\theta\\mathcal L$.\nThe loss is a [STATE] diagnostic, not a parameter. Its definition (cross-entropy) is a [CONFIG] modeling choice; its value is recomputed every step from outputs and targets. It is the scalar the optimizer descends.\n5. Perplexity A loss in nats is the right quantity to optimize but an awkward one to talk about. The metric usually reported instead simply undoes the logarithm, turning the loss into a count of effective choices that a human can reason about directly.\nThe standard human-facing metric exponentiates the per-token cross-entropy:\n$$ \\mathrm{PPL}=\\exp\\!\\big(\\mathcal L\\big)=\\exp\\!\\Big(-\\tfrac1n\\textstyle\\sum_t\\log\\hat p_{t,x_t}\\Big). $$Interpretation: PPL is the effective branching factor — the size of a uniform distribution with the same per-token uncertainty. PPL $=1$ is perfect determinism; uniform guessing over $V$ tokens gives PPL $=V$. Because $\\mathrm{PPL}=e^{\\mathcal L}$ is monotonically increasing in $\\mathcal L$, minimizing loss minimizes perplexity; we report PPL only because \u0026ldquo;choosing among ~8 options\u0026rdquo; is more legible than \u0026ldquo;2.08 nats.\u0026rdquo;\nWorked example. True-token probabilities $0.6,0.2,0.1$ over three positions:\n$$ \\mathcal L=\\tfrac13(-\\log0.6-\\log0.2-\\log0.1)=\\tfrac13(0.511+1.609+2.303)=1.474\\ \\text{nats},\\quad \\mathrm{PPL}=e^{1.474}=4.37. $$Sharpen the predictions to $0.9,0.8,0.7$: $\\mathcal L=0.267$, $\\mathrm{PPL}=1.31$. Note PPL\u0026rsquo;s sensitivity — modest probability gains compress the branching factor sharply, which is why small loss improvements correspond to meaningful capability gains.\nCaveat. PPL is only comparable across models that share a tokenizer: a coarser tokenization packs more characters per token, lowering per-token PPL without any modeling improvement. Cross-tokenizer comparisons should use bits-per-byte instead.\nLearned / config / state Quantity Category Choice of loss (cross-entropy) [CONFIG] Label smoothing / soft targets [CONFIG] $\\mathcal L$ value, $\\hat{\\mathbf p}$, PPL [STATE] Figure 3. Small gains in probability, large gains in branching factor.\nConclusion Cross-entropy $H(p,\\hat p)=H(p)+D_{\\mathrm{KL}}(p\\|\\hat p)$ shows training to be minimization of the KL divergence from the model to the true language distribution, floored by the irreducible entropy $H(p)$. With empirical one-hot targets it collapses to $-\\log\\hat p_{x_t}$, a proper scoring rule whose unbounded penalty for confident errors forces calibration — the reason we train on cross-entropy rather than accuracy. Measured in bits it equals the optimal compression rate, linking modeling to compression. Perplexity $e^{\\mathcal L}$ re-expresses the loss as an effective branching factor (comparable only within a fixed tokenizer). The loss is transient state whose definition is a config choice; its logit-gradient $\\hat{\\mathbf p}-\\mathbf y$ is the seed for the next lesson.\nWe have now defined what \u0026ldquo;good\u0026rdquo; means and how to measure it. Everything up to here has been the forward direction — \u0026ldquo;inference reads.\u0026rdquo; With a differentiable loss and the seed gradient $\\hat{\\mathbf p}-\\mathbf y$ in hand, the rest of the course is the backward direction, where \u0026ldquo;training writes.\u0026rdquo; Next: how that seed is propagated to a gradient over all $P$ parameters, and how the parameters are updated.\n← Lesson 5 — Output Head \u0026amp; Objective · Course index · Next: Lesson 7 — Backpropagation \u0026amp; Gradient Descent → Appendices: A — Acronyms · B — Formulas · C — Attention Variants The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/tutorials/llm-lessons/lesson-06-the-loss-function/","summary":"\u003ch1 id=\"lesson-6--the-loss-function\"\u003eLesson 6 — The Loss Function\u003c/h1\u003e\n\u003cp\u003e\u003cem\u003eTechnical course on LLM mechanics — Lesson 6 of 10.\u003c/em\u003e\u003c/p\u003e\n\u003chr\u003e\n\u003ch2 id=\"recap-of-lesson-5\"\u003eRecap of Lesson 5\u003c/h2\u003e\n\u003cp\u003eThe head produces logits $\\mathbf z=\\mathbf h^\\top W_U$ and softmax gives $\\hat{\\mathbf p}\\in\\Delta^{V-1}$. The objective is maximum likelihood of observed tokens = minimum average cross-entropy $\\mathcal L=-\\frac1n\\sum_t\\log\\hat p_{t,x_t}$, self-supervised because the label at $t$ is $x_{t+1}$. The softmax+cross-entropy pairing yields the clean gradient $\\partial\\mathcal L/\\partial\\mathbf z=\\hat{\\mathbf p}-\\mathbf y$. This lesson develops the loss information-theoretically, derives its key properties, and connects it to perplexity.\u003c/p\u003e","title":"The Loss Function"},{"content":"Chapter 6 — The Translators: Encoder–Decoder Models We have met the writer, who generates text from left to right, and the reader, who understands text in a single bidirectional pass. Now consider a job that seems to need both at once: translation. To translate an English sentence into German, you must first read the English completely — understanding it fully, including how its ending clarifies its beginning — and then write a fresh German sentence based on that understanding. Summarization is the same shape: read a long article thoroughly, then write a short summary. Answering a question about a passage is the same shape again. These \u0026ldquo;read one thing, write a different thing\u0026rdquo; tasks have a name — sequence-to-sequence — and they are the natural home of the encoder–decoder model, the family that includes T5 and BART.\nWhy these tasks want two stacks What makes translation and summarization different from plain storytelling is an asymmetry. The input is complete and available from the very start, and it never changes. The output, by contrast, has to be built up one word at a time. You have a finished thing to read and an unfinished thing to write, and they are not the same length or even the same language. That mismatch is the whole reason a special architecture exists.\nThe natural response is to split the work between two specialists. An encoder — a reader, exactly the bidirectional reader from the last chapter — digests the entire input and turns it into a rich internal memory. A decoder — a writer, exactly the left-to-right storyteller from Chapter 4 — produces the output one word at a time, and at every step it consults that memory. This mirrors what a human translator actually does: read the whole source sentence carefully first, then start writing the translation, glancing back at the original as needed. You would never translate word by word having read only the opening; you read it all, then write.\nSplitting the work this way buys three concrete things. The reading happens bidirectionally, so the input is understood fully. The reading happens once, and the resulting memory is reused for every output word, which is efficient. And the two roles are decoupled, so the reader can specialize in understanding while the writer specializes in producing fluent output.\nThe one sentence that captures the whole architecture Here is the single most useful thing to hold in your head about this entire model family: the encoder compresses the context, the decoder generates conditioned on it, and the bridge between them — called cross-attention — is the only genuinely new mechanism. Almost everything else you would need to understand an encoder–decoder model, you already know. The map of meaning, attention, the towers of blocks, the output head, the training game — all of it is borrowed, essentially unchanged, from the writer and reader we have already met. Only one new piece is added. So if we understand that one new piece, we understand the whole thing.\nCross-attention: the only new idea The reader produces its memory — a set of meaning-fingerprints, one per input word. The writer then generates the output one word at a time. The new question is: how does the writer, while composing each word, draw on the reader\u0026rsquo;s memory of the input? The answer is cross-attention, and it is just a small variation on the attention you already understand.\nRecall the party from Chapter 3: each word poses a question, every word offers a label and some information, and matches let information flow. Ordinary attention has a sentence asking questions of itself. Cross-attention changes exactly one thing: the questions come from the writer (the half-finished output), while the labels and information come from the reader (the memory of the input). That is the entire difference — a change of who asks and who answers.\nPicture the human translator again, partway through a German sentence. To choose her next word, she asks herself, \u0026ldquo;which part of the English source do I need right now?\u0026rdquo; She scans the source, her eye lands most heavily on the relevant word or two, and she pulls their meaning across to shape what she writes next. Cross-attention is exactly that scan. As the writer composes each output word, it poses a question, surveys the reader\u0026rsquo;s memory of the input, finds the input words that best answer it, and draws their meaning in. The pattern of where its attention lands — strong on this source word, weak on that one — amounts to a soft, automatic alignment between the output and the input. For translation it tends to light up right where each output word corresponds to a source word. This learned, flexible alignment is what replaced the rigid, hand-built translation tables of older systems.\nTwo details are worth noting. Cross-attention has no \u0026ldquo;no peeking\u0026rdquo; rule — the writer may look at any part of the input, including parts that correspond to later positions, because the whole input is already known and there is nothing to spoil. And cross-attention is the only channel through which the writer ever draws on the input. Cut it, and the writer would simply ignore the source and ramble on its own. It is the bond that makes the two stacks one machine.\nFigure 1. Cross-attention: the questions change hands.\nThree designs on a single spectrum A natural objection arises here. Modern chatbots are decoder-only — pure storytellers — and yet they translate and summarize perfectly well, just by being given the input as part of their prompt and continuing from it. So do we really need two separate stacks?\nThe elegant answer is that all of these designs are points on a single spectrum, and the dial that distinguishes them is just one rule: what is each word allowed to look at? At one end sits the encoder-only reader, where every word sees every other word — a superb understander that cannot generate. At the other end sits the decoder-only writer, where every word sees only what came before — a superb generator that reads its input merely as more text to continue. The encoder–decoder sits in between by a different route: it gives reading and writing their own separate stacks, bonded by cross-attention. And there is an even leaner middle option, sometimes called a prefix language model, which uses a single stack but lets the input portion look around freely while keeping the output portion strictly left-to-right — getting the main benefit of bidirectional reading without a second stack at all.\nReading across that spectrum, the looking rule gets steadily stricter and the way the input gets folded in gets steadily more implicit. This is the deep reason a decoder-only chatbot can translate at all: a left-to-right stack plus a well-crafted prompt approximates the prefix model, which approximates the full encoder–decoder. The field largely consolidated on decoder-only models for general-purpose use because one uniform stack is simpler to scale up. But the dedicated two-stack design persists wherever reading the input really well is decisive — and controlled experiments have found it genuinely strongest for translation and summarization.\nFigure 2. One dial: what is each word allowed to look at?.\nHow they learn: repairing damaged text To train these models on mountains of unlabeled text, you need a task that manufactures its own answers. The trick both T5 and BART use is denoising: take a piece of clean text, deliberately damage it, feed the damaged version to the reader, and train the writer to reconstruct what was lost. The damage itself creates a matched input-output pair out of a single document — no human labeling required.\nT5\u0026rsquo;s version is a fill-in-the-blanks puzzle on a larger scale: blank out a few short stretches of words, mark each with a placeholder, and have the writer produce just the missing pieces. Because the writer only has to generate the small missing bits rather than the entire document, training stays cheap, and because a blank\u0026rsquo;s contents depend on words on both sides, it forces genuinely bidirectional reading. BART\u0026rsquo;s version is a grab-bag of corruptions — masking words, deleting words so the model must work out where something went missing, shuffling sentence order, and more — with the writer rebuilding the whole original. The variety is the point: learning to undo many kinds of damage builds exactly the robust reading and fluent writing that translation and summarization demand.\nOne subtlety connects back to cross-attention. Only the writer produces the final output and therefore the only direct measure of error. So how does the reader ever improve? Entirely through cross-attention, which is the single channel carrying the learning signal back into the reader. The two stacks are trained together as one, and over time the reader learns to produce exactly the kind of memory the writer keeps reaching for. (The general mechanics of how this learning signal flows are the subject of Chapter 9 .)\nUsing a translator: read once, write many When a trained encoder–decoder model is put to work, the division of labor is clean and efficient. The reader runs exactly once on the input, producing the memory. Then the writer generates the output one word at a time, each step consulting that same fixed memory. The expensive reading is paid for once and reused for every output word.\nBecause tasks like translation have a roughly correct answer — unlike open-ended creative writing — these models typically choose their words a little more carefully than a freewheeling chatbot. A common method keeps several promising partial sentences alive in parallel and lets the best overall one win, rather than greedily grabbing the single most likely next word and risking painting itself into a corner. There are also characteristic failure modes to manage: because the writer freely re-consults the input without remembering what it has already covered, it can fixate on one source phrase and repeat it, or skip another and drop it. Techniques that track how much of the source has been \u0026ldquo;covered\u0026rdquo; help discourage both. And for things that must be reproduced exactly — a name, a number, a rare word — the model can be given the ability to copy a word straight from the source rather than trying to regenerate it from scratch.\nFigure 3. Read once, write many.\nT5\u0026rsquo;s big idea, and how T5 and BART differ T5 is built around one organizing insight: every language task can be reshaped as \u0026ldquo;text in, text out,\u0026rdquo; which is exactly the shape an encoder–decoder already has. So rather than bolting different machinery onto the model for each task, you simply describe the task in plain words at the front of the input and read the answer as text. Prefix the input with \u0026ldquo;translate English to German:\u0026rdquo; and the output is the translation. Prefix it with \u0026ldquo;summarize:\u0026rdquo; and the output is a summary. Even a yes-or-no classification becomes \u0026ldquo;generate the word yes or no,\u0026rdquo; and a numeric score becomes \u0026ldquo;generate the number as text.\u0026rdquo; One model, one setup, one method, handling everything — only the wording of the input and the training data change.\nAs for the two famous names: both are encoder–decoder models sharing the identical cross-attention core, and they differ mostly in their pretraining damage and minor engineering choices. BART is essentially the classic original design, taught to repair messy text through its grab-bag of corruptions, and it is especially strong at summarization. T5 keeps the same skeleton but reframes everything as text-to-text and trains with fill-in-the-blank span corruption. Think of BART as \u0026ldquo;the classic translator, taught to fix damaged text\u0026rdquo; and T5 as \u0026ldquo;the same machine, taught that every task is just text-in, text-out.\u0026rdquo; The mechanism that makes them work is the same in both — and, remarkably, the same idea reappears far beyond text, bonding a reader and a writer in systems that turn speech into transcripts or images into captions.\nWhat to carry forward An encoder–decoder model bolts a reader to a writer for the \u0026ldquo;read one thing, write a different thing\u0026rdquo; jobs of translation and summarization. The reader compresses the input into a memory; the writer generates the output while consulting that memory through cross-attention — the one genuinely new mechanism, which is just attention with the questions coming from the writer and the answers from the reader. All three language-model shapes — reader, writer, and the two-stack translator — sit on one spectrum defined by what each word is allowed to see.\nSo far, every model we have met uses all of itself to process every word. The next chapter asks a radical question: what if a model could be enormous in what it knows, yet only switch on a small part of itself for any given word? That is the idea behind mixture-of-experts, in Chapter 7 .\n← Chapter 5 · The Readers · Contents · Chapter 7 · The Committee → The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/tutorials/inside-the-machines/chapter-06-encoder-decoder-models/","summary":"\u003ch1 id=\"chapter-6--the-translators-encoderdecoder-models\"\u003eChapter 6 — The Translators: Encoder–Decoder Models\u003c/h1\u003e\n\u003cp\u003eWe have met the writer, who generates text from left to right, and the reader, who understands text in a single bidirectional pass. Now consider a job that seems to need both at once: translation. To translate an English sentence into German, you must first \u003cem\u003eread\u003c/em\u003e the English completely — understanding it fully, including how its ending clarifies its beginning — and then \u003cem\u003ewrite\u003c/em\u003e a fresh German sentence based on that understanding. Summarization is the same shape: read a long article thoroughly, then write a short summary. Answering a question about a passage is the same shape again. These \u0026ldquo;read one thing, write a different thing\u0026rdquo; tasks have a name — \u003cem\u003esequence-to-sequence\u003c/em\u003e — and they are the natural home of the \u003cem\u003eencoder–decoder\u003c/em\u003e model, the family that includes T5 and BART.\u003c/p\u003e","title":"The Translators: Encoder–Decoder Models"},{"content":"Lesson 6 — The Network: U-Net and Time Conditioning Recap of Lesson 5. The $\\epsilon$-parameterization $\\mu_\\theta = \\frac{1}{\\sqrt{\\alpha_t}}(x_t - \\frac{\\beta_t}{\\sqrt{1-\\bar\\alpha_t}}\\epsilon_\\theta)$ makes schedule factors cancel, collapsing the variational objective into the simple loss $L_{\\text{simple}} = \\mathbb E_{t,x_0,\\epsilon}\\,\\|\\epsilon - \\epsilon_\\theta(\\sqrt{\\bar\\alpha_t}x_0+\\sqrt{1-\\bar\\alpha_t}\\epsilon,\\,t)\\|^2$. The network is a noise predictor with unlimited, cheaply-generated training pairs. This lesson opens the box: what is $\\epsilon_\\theta$ actually made of?\nWe have spent five lessons describing a job without once meeting the worker who does it: take a noisy image, hand back the noise hidden inside it, and do so at every level of corruption with the same pair of hands. This lesson opens the box we have only ever named by its task. Inside is not a featureless slab but a particular, well-argued shape — a network that narrows until it can see the whole picture at a glance, widens again to put the grain back, and ferries fine detail across the gap on dedicated wires — and that is told, every time, exactly how noisy the image before it is.\nOverview $\\epsilon_\\theta$ must map an $H\\times W\\times C$ tensor to an $H\\times W\\times C$ tensor of the same shape — it is a dense prediction network, not a classifier — while reasoning across spatial scales, since noise corrupts fine texture and coarse structure alike. The U-Net satisfies both constraints: a contracting encoder builds global context at low resolution, an expanding decoder restores spatial precision, and skip connections carry high-frequency detail across the bottleneck. A single shared $\\theta$ must denoise at every noise level, so the network is additionally conditioned on $t$ through a sinusoidal embedding fed into a small MLP and broadcast into every block as a learned scale-and-shift. We make the block composition, attention placement and cost, time-conditioning equations, and the U-Net-vs-DiT distinction precise, with correct PyTorch-style code and tensor shapes.\nWhy image-to-image, and why multi-scale The target $\\epsilon$ lives in the same space as $x_t$ — one value per pixel-channel — so $\\epsilon_\\theta$ resembles a segmentation network, not a classifier. Denoising is inherently multi-scale: recovering a face needs local cues (edges, grain) and global ones (this region is an eye, hence dark). The contracting path captures global context cheaply at low resolution; the expanding path restores spatial precision; skip connections prevent fine detail from being destroyed at the bottleneck. The same translation-equivariance argument that motivates convolutions for natural images applies here.\nFigure 1. Why attention is only affordable once the map is small.\nAnatomy of the U-Net A diffusion U-Net is a stack of stages at decreasing then increasing resolution:\nEncoder (downsampling path). Repeated residual blocks followed by downsampling. Channel width grows as resolution shrinks to keep per-stage compute and representational capacity roughly balanced, e.g. $256^2\\!\\times\\!128 \\to 128^2\\!\\times\\!256 \\to 64^2\\!\\times\\!512 \\to 32^2\\!\\times\\!512$. A common width rule is a base channel count $C_0$ times a tuple of multipliers, e.g. $C_0\\cdot(1,2,4,4)$. Bottleneck. Residual + self-attention + residual at the lowest resolution, where the receptive field covers the whole image at low token count. Decoder (upsampling path). Mirror of the encoder; each stage upsamples and concatenates the matching encoder feature map via a skip connection before its residual blocks (so the first conv of a decoder block sees $2\\times$ the channels). Output head. GroupNorm → SiLU → final $3\\times3$ conv to $C$ channels — the predicted $\\hat\\epsilon$ (or $\\hat x_0$, or $v$). This last conv is often zero-initialized so the network starts as the identity-noise predictor. Residual blocks. Each block is GroupNorm → SiLU → Conv(3×3) → (add time projection) → GroupNorm → SiLU → Dropout → Conv(3×3), with the input added back through a skip (a $1\\times1$ conv on the skip if channel counts differ):\n$$ h_1=\\mathrm{Conv}_1(\\mathrm{SiLU}(\\mathrm{GN}(h))),\\qquad h_2=\\mathrm{Conv}_2(\\mathrm{SiLU}(\\mathrm{GN}(h_1+\\text{time term}))),\\qquad \\mathrm{out}=h_2+\\mathrm{Skip}(h). $$GroupNorm (not BatchNorm) is used because diffusion batches are small and the statistics must be batch-independent; SiLU/Swish is the standard activation. Residual connections keep gradients healthy through depth.\nSelf-attention placement and cost. Convolutions have a bounded receptive field; self-attention lets distant regions coordinate, which is necessary for globally coherent samples. Attention over a feature map of $N=HW$ spatial positions forms an $N\\times N$ score matrix, so its time and memory cost is\n$$ O\\big((HW)^2\\,d\\big), $$quadratic in the number of pixels. This is why attention is inserted only at coarse resolutions (e.g. $32^2$, $16^2$, $8^2$), where $HW$ is small; placing it at $256^2$ would cost $(256^2)^2\\approx 4.3\\times 10^9$ score entries per head. Improved diffusion architectures (Nichol \u0026amp; Dhariwal 2021; Dhariwal \u0026amp; Nichol 2021) use multi-head attention at multiple low resolutions, more residual blocks per stage, and BigGAN-style up/down residual blocks, finding these scale the model favorably.\nDown/upsampling choices. Downsampling is a stride-2 convolution (learned) or average pooling; upsampling is nearest-neighbor interpolation followed by a $3\\times3$ conv (preferred over transposed conv, which produces checkerboard artifacts). Parameter count is dominated by the wide low-resolution stages and the bottleneck attention; FLOPs are dominated by the high-resolution stages, motivating the channel-growth schedule above.\nFigure 2. Narrow to see everything, widen to put the grain back.\nInjecting the timestep $t$ A single set of weights must serve a thousand different noise levels, which raises the obvious question: how does the network know which level it is looking at?\nThe shared weights must denoise at every noise level, so the network needs to know $t$. We do not feed the raw integer; we build a smooth, high-dimensional embedding.\nStep 1 — sinusoidal embedding. Map the scalar $t$ to a vector of sines and cosines at geometrically spaced frequencies (Vaswani et al., 2017):\n$$ \\text{emb}(t)_{2i} = \\sin\\!\\left(\\frac{t}{10000^{2i/d}}\\right),\\qquad \\text{emb}(t)_{2i+1} = \\cos\\!\\left(\\frac{t}{10000^{2i/d}}\\right),\\qquad i=0,\\dots,\\tfrac{d}{2}-1. $$The geometric frequency ladder spans wavelengths from $2\\pi$ to $2\\pi\\cdot 10000$, so coarse and fine differences in $t$ are simultaneously representable. The construction is motivated by linearity: for a fixed offset $\\delta$, $\\text{emb}(t+\\delta)$ is a fixed linear (rotation) function of $\\text{emb}(t)$, since $\\sin(\\omega(t+\\delta))$ and $\\cos(\\omega(t+\\delta))$ are linear combinations of $\\sin(\\omega t),\\cos(\\omega t)$. The network can therefore read off relative noise level via a linear map, and nearby timesteps map to nearby vectors (Lipschitz in $t$), which raw one-hot or integer inputs would not provide.\nStep 2 — small MLP. Pass $\\text{emb}(t)$ through a two-layer MLP with a nonlinearity to produce the conditioning vector $\\tau \\in \\mathbb R^{d_\\tau}$:\n$$ \\tau = W_2\\,\\mathrm{SiLU}(W_1\\,\\text{emb}(t) + b_1) + b_2. $$Step 3 — FiLM / adaLN scale-shift injection. Inside each residual block, $\\tau$ is projected to the block\u0026rsquo;s channel count $C$ and used to modulate the normalized features. The simplest form is an additive bias (FiLM with unit scale; Perez et al., 2018), $h\\leftarrow h + W_\\tau\\tau$, broadcast over $H,W$. The more expressive and now-standard form is a per-channel scale-and-shift applied to a normalized activation:\n$$ \\boxed{\\ (\\gamma,\\beta)=W_\\tau\\,\\tau,\\qquad h \\leftarrow (1+\\gamma)\\odot \\mathrm{Norm}(h) + \\beta\\ } $$with $\\gamma,\\beta\\in\\mathbb R^{C}$ broadcast over spatial dimensions. This is FiLM/adaptive-GroupNorm in the U-Net and adaLN in transformers. Writing $1+\\gamma$ rather than $\\gamma$ centers the modulation at identity, so zero-initializing $W_\\tau$ makes each block start as an unconditioned residual block and learn conditioning gradually (the \u0026ldquo;Zero\u0026rdquo; variant; see DiT below).\nA more complete forward-pass sketch import math, torch, torch.nn as nn, torch.nn.functional as F def sinusoidal_embedding(t, dim): # t: (B,) integer or float timesteps -\u0026gt; (B, dim) half = dim // 2 freqs = torch.exp(-math.log(10000) * torch.arange(half, device=t.device) / half) args = t[:, None].float() * freqs[None, :] # (B, half) return torch.cat([torch.sin(args), torch.cos(args)], dim=-1) # (B, dim) class ResBlock(nn.Module): def __init__(self, c_in, c_out, t_dim): super().__init__() self.norm1 = nn.GroupNorm(32, c_in) self.conv1 = nn.Conv2d(c_in, c_out, 3, padding=1) self.t_proj = nn.Linear(t_dim, 2 * c_out) # -\u0026gt; (gamma, beta) self.norm2 = nn.GroupNorm(32, c_out) self.conv2 = nn.Conv2d(c_out, c_out, 3, padding=1) self.skip = nn.Conv2d(c_in, c_out, 1) if c_in != c_out else nn.Identity() def forward(self, h, tau): # h:(B,c_in,H,W) tau:(B,t_dim) x = self.conv1(F.silu(self.norm1(h))) # (B,c_out,H,W) gamma, beta = self.t_proj(tau).chunk(2, dim=1) # each (B,c_out) x = self.norm2(x) x = (1 + gamma[:, :, None, None]) * x + beta[:, :, None, None] # adaGN scale-shift x = self.conv2(F.silu(x)) # (B,c_out,H,W) return x + self.skip(h) class AttnBlock(nn.Module): # self-attention over HW tokens def __init__(self, c, heads=4): super().__init__() self.norm = nn.GroupNorm(32, c) self.qkv = nn.Conv2d(c, 3 * c, 1) self.proj = nn.Conv2d(c, c, 1) self.heads = heads def forward(self, h): # h:(B,C,H,W) B, C, H, W = h.shape q, k, v = self.qkv(self.norm(h)).chunk(3, dim=1) # each (B,C,H,W) # reshape to (B, heads, HW, C/heads) resh = lambda z: z.reshape(B, self.heads, C // self.heads, H * W).transpose(-1, -2) out = F.scaled_dot_product_attention(resh(q), resh(k), resh(v)) # O((HW)^2) out = out.transpose(-1, -2).reshape(B, C, H, W) return h + self.proj(out) # residual class UNet(nn.Module): def __init__(self, c_img=3, c0=128, mults=(1, 2, 4, 4), t_dim=512, attn_res={16, 8}): super().__init__() self.t_mlp = nn.Sequential(nn.Linear(c0, t_dim), nn.SiLU(), nn.Linear(t_dim, t_dim)) self.c0 = c0 # ... encoder ResBlocks/AttnBlocks/Downsample, bottleneck, decoder mirrored ... # encoder stages append feature maps to `skips`; decoder concatenates them. self.in_conv = nn.Conv2d(c_img, c0, 3, padding=1) self.out = nn.Sequential(nn.GroupNorm(32, c0), nn.SiLU(), nn.Conv2d(c0, c_img, 3, padding=1)) # zero-init last conv def forward(self, x_t, t): # x_t:(B,C,H,W) t:(B,) tau = self.t_mlp(sinusoidal_embedding(t, self.c0)) # (B, t_dim) h = self.in_conv(x_t) skips = [h] # for stage in encoder: h = resblock(h, tau); (attn if H in attn_res); skips.append(h); h = downsample(h) # h = bottleneck_res2(attn(bottleneck_res1(h, tau)), tau) # for stage in decoder: h = upsample(h); h = cat([h, skips.pop()], 1); h = resblock(h, tau) return self.out(h) # (B,C,H,W) predicted noise scaled_dot_product_attention makes the $O((HW)^2)$ cost explicit and uses a fused/flash kernel when available. Use GroupNorm(32, c) with c divisible by 32; zero-initialize the final conv and each block\u0026rsquo;s t_proj for stable early training. Skips must be popped in the reverse order they were pushed; mismatched resolutions are the most common U-Net bug.\nU-Net vs. Diffusion Transformer (DiT) None of the framework so far cares that the worker happens to be a U-Net; to prove the point, we now swap the worker out entirely and watch the loss and loop stay put.\nThe framework of Lessons 1–5 is architecture-agnostic — only $\\epsilon_\\theta$ changes. The Diffusion Transformer (Peebles \u0026amp; Xie, 2023) replaces the convolutional U-Net with a plain transformer on a (latent) image:\nPatchify. Split the $h\\times w\\times c$ latent into non-overlapping $p\\times p$ patches and linearly embed each into a $D$-dimensional token, giving $N=(h/p)(w/p)$ tokens. Smaller $p$ means more tokens, higher cost, finer detail. Positional + time embeddings. Add (sinusoidal or learned) 2-D positional embeddings to the tokens. The timestep $t$ (and any class/text condition $c$) are embedded and supplied not as tokens but as conditioning to adaLN. adaLN-Zero. Each transformer block replaces LayerNorm\u0026rsquo;s affine parameters with $\\gamma,\\beta$ regressed from $\\tau$ (the embedded $t,c$), exactly the scale-shift of the U-Net but on token features, plus a learned residual gate $\\alpha$: $h\\leftarrow h+\\alpha\\odot\\mathrm{Block}((1+\\gamma)\\,\\mathrm{LN}(h)+\\beta)$. The gate and the modulation MLP are zero-initialized so each block starts as the identity — the \u0026ldquo;Zero\u0026rdquo; in adaLN-Zero — which is what makes deep DiTs trainable. Unpatchify. A final linear layer maps each token back to a $p\\times p\\times c$ patch; reassemble into the output grid. Same loss, same training loop, same $(x_t,t)\\mapsto\\hat\\epsilon$ signature. DiT trades the U-Net\u0026rsquo;s built-in multi-scale inductive bias (downsampling + skips) for global self-attention at every layer and clean scaling with compute; it is the backbone of recent large text-to-image and video systems. Both designs can be combined with EDM-style preconditioning (Karras et al., 2022), which rescales the network\u0026rsquo;s input, output, and skip so it always solves a unit-variance regression regardless of noise level (cf. Lesson 5\u0026rsquo;s target-variance remark).\nFigure 3. Swap the worker; the loss and the loop stay put.\nExercises 1. Why attention only at coarse resolutions. Self-attention over an $H\\times W$ feature map forms an $N\\times N$ score matrix with $N=HW$, costing $O((HW)^2)$. Compute the ratio of score-matrix entries between a $256\\times256$ map and a $32\\times32$ map, and use it to justify placing attention only at low resolution.\n$\\dfrac{(256^2)^2}{(32^2)^2}=\\left(\\dfrac{256}{32}\\right)^4=8^4=4096$. The full-resolution map needs $\\sim\\!4096\\times$ as many score entries (and as much attention compute/memory) as the $32^2$ map — prohibitive — which is why attention is inserted only where $HW$ is already small.\n2. Linearity of the sinusoidal embedding. Show that for a fixed offset $\\delta$, the pair $(\\sin\\omega t,\\cos\\omega t)$ is mapped to $(\\sin\\omega(t+\\delta),\\cos\\omega(t+\\delta))$ by a rotation matrix that does not depend on $t$.\nBy the angle-addition formulas,\n$$ \\begin{pmatrix}\\sin\\omega(t+\\delta)\\\\ \\cos\\omega(t+\\delta)\\end{pmatrix} =\\begin{pmatrix}\\cos\\omega\\delta \u0026 \\sin\\omega\\delta\\\\ -\\sin\\omega\\delta \u0026 \\cos\\omega\\delta\\end{pmatrix} \\begin{pmatrix}\\sin\\omega t\\\\ \\cos\\omega t\\end{pmatrix}. $$The matrix depends only on $\\omega\\delta$, not on $t$ — so $\\text{emb}(t+\\delta)$ is a fixed linear (rotation) function of $\\text{emb}(t)$, letting the network read relative timestep shifts with a linear map.\n3. Identity-at-init modulation. Explain why the scale-shift is written $(1+\\gamma)\\odot\\mathrm{Norm}(h)+\\beta$ rather than $\\gamma\\odot\\mathrm{Norm}(h)+\\beta$, and what zero-initializing the projection that produces $(\\gamma,\\beta)$ achieves.\nWith the projection zero-initialized, $\\gamma=\\beta=0$, so the modulation becomes $(1+0)\\odot\\mathrm{Norm}(h)+0=\\mathrm{Norm}(h)$ — the identity transform. The block therefore starts as an ordinary, unconditioned residual block and learns conditioning gradually, which stabilizes early training (the \u0026ldquo;Zero\u0026rdquo; in adaLN-Zero). Writing plain $\\gamma$ would zero out the activations at init instead.\n4. Decoder channel doubling. Why does the first convolution of a decoder stage receive $2C$ input channels when the stage works at width $C$?\nEach decoder stage concatenates the matching encoder feature map (via the skip connection) with the upsampled decoder feature along the channel axis — $C$ channels from each — giving $2C$. The first conv must therefore accept $2C$ inputs. (Popping skips in the wrong order, or mismatched resolutions, is the most common U-Net bug.)\nConclusion The noise predictor $\\epsilon_\\theta$ is an image-to-image U-Net: residual blocks of GroupNorm → SiLU → Conv with adaptive scale-shift time injection, self-attention at coarse resolutions (paying its $O((HW)^2)$ cost only where $HW$ is small), learned down/up-sampling, and skip connections matched to the multi-scale nature of denoising. The timestep enters as a sinusoidal embedding → MLP → per-block FiLM/adaLN scale-shift, justified by the linearity of sinusoids under timestep offsets, so one shared network denoises at every level. The DiT keeps the loss and loop but swaps the convolutional backbone for a transformer with patchify, positional+time embeddings, adaLN-Zero conditioning, and unpatchify. We now have the loss (Lesson 5) and the machine that computes it (this lesson). In Lesson 7 we assemble them into the complete, runnable training algorithm.\nThe box is open, and the worker turns out to be a sensible machine rather than a mystery. We now have the objective (Lesson 5) and the apparatus that computes it (this lesson). Lesson 7 puts the two on the same bench and writes the loop that actually trains them.\nReferences O. Ronneberger, P. Fischer, T. Brox. U-Net: Convolutional Networks for Biomedical Image Segmentation. MICCAI 2015. (Encoder–decoder with skip connections.) J. Ho, A. Jain, P. Abbeel. Denoising Diffusion Probabilistic Models. NeurIPS 2020. (U-Net backbone with sinusoidal time embedding and attention.) A. Nichol, P. Dhariwal. Improved Denoising Diffusion Probabilistic Models. ICML 2021. (Architecture/attention/cosine-schedule improvements.) P. Dhariwal, A. Nichol. Diffusion Models Beat GANs on Image Synthesis. NeurIPS 2021. (Improved U-Net: multi-resolution attention, more blocks, BigGAN up/down.) A. Vaswani, et al. Attention Is All You Need. NeurIPS 2017. (Sinusoidal positional embeddings; scaled dot-product attention.) E. Perez, F. Strub, H. de Vries, V. Dumoulin, A. Courville. FiLM: Visual Reasoning with a General Conditioning Layer. AAAI 2018. (Feature-wise scale-shift conditioning.) W. Peebles, S. Xie. Scalable Diffusion Models with Transformers (DiT). ICCV 2023. (Patchify, adaLN-Zero, unpatchify.) T. Karras, M. Aittala, T. Aila, S. Laine. Elucidating the Design Space of Diffusion-Based Generative Models (EDM). NeurIPS 2022. (Input/output/skip preconditioning.) ← Previous: Lesson 5 — The Simplified Objective · Course Index · Next: Lesson 7 — The Training Algorithm → · Appendix A: Acronyms · Appendix B: Formulas The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/tutorials/diffusion-lessons/lesson-06-unet-architecture/","summary":"\u003ch1 id=\"lesson-6--the-network-u-net-and-time-conditioning\"\u003eLesson 6 — The Network: U-Net and Time Conditioning\u003c/h1\u003e\n\u003cblockquote\u003e\n\u003cp\u003e\u003cstrong\u003eRecap of Lesson 5.\u003c/strong\u003e The $\\epsilon$-parameterization $\\mu_\\theta = \\frac{1}{\\sqrt{\\alpha_t}}(x_t - \\frac{\\beta_t}{\\sqrt{1-\\bar\\alpha_t}}\\epsilon_\\theta)$ makes schedule factors cancel, collapsing the variational objective into the simple loss $L_{\\text{simple}} = \\mathbb E_{t,x_0,\\epsilon}\\,\\|\\epsilon - \\epsilon_\\theta(\\sqrt{\\bar\\alpha_t}x_0+\\sqrt{1-\\bar\\alpha_t}\\epsilon,\\,t)\\|^2$. The network is a noise predictor with unlimited, cheaply-generated training pairs. This lesson opens the box: what is $\\epsilon_\\theta$ actually made of?\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003chr\u003e\n\u003cblockquote\u003e\n\u003cp\u003e\u003cem\u003eWe have spent five lessons describing a job without once meeting the worker who does it: take a noisy image, hand back the noise hidden inside it, and do so at every level of corruption with the same pair of hands. This lesson opens the box we have only ever named by its task. Inside is not a featureless slab but a particular, well-argued shape — a network that narrows until it can see the whole picture at a glance, widens again to put the grain back, and ferries fine detail across the gap on dedicated wires — and that is told, every time, exactly how noisy the image before it is.\u003c/em\u003e\u003c/p\u003e","title":"U-Net Architecture"},{"content":"Lesson 7 — Backpropagation \u0026amp; Gradient Descent Technical course on LLM mechanics — Lesson 7 of 10.\nRecap of Lesson 6 Cross-entropy $=H(p)+D_{\\mathrm{KL}}(p\\|\\hat p)$, so minimizing it minimizes KL to the true distribution, floored by language\u0026rsquo;s entropy $H(p)$. With one-hot targets it is $-\\log\\hat p_{x_t}$, a proper scoring rule (hence calibration, hence why we don\u0026rsquo;t train on accuracy). Perplexity $e^{\\mathcal L}$ is the effective branching factor. The logit gradient is $\\partial\\mathcal L/\\partial\\mathbf z=\\hat{\\mathbf p}-\\mathbf y$. This lesson turns that single gradient into $\\nabla_\\theta\\mathcal L\\in\\mathbb R^P$ and specifies the update — the precise mechanics of what gets written.\nWe have spent five lessons opening the box and one lesson defining what \u0026ldquo;good\u0026rdquo; means. We now have a number, $\\mathcal L$, that scores the model, and a single derivative at the output. What we still lack is the machine that converts that score into a change in every one of the model\u0026rsquo;s parameters. This lesson and the next are the engine room: here we compute the gradient that says which way to move every weight, and Lesson 8 takes the step. This is the first lesson where $\\theta$ actually changes — the concrete content of \u0026ldquo;training writes.\u0026rdquo;\n1. The optimization view $\\mathcal L:\\mathbb R^P\\to\\mathbb R$ is a scalar field over parameter space. The gradient\n$$ \\nabla_\\theta\\mathcal L=\\Big(\\tfrac{\\partial\\mathcal L}{\\partial\\theta_1},\\dots,\\tfrac{\\partial\\mathcal L}{\\partial\\theta_P}\\Big) $$points in the direction of steepest ascent (first-order Taylor: $\\mathcal L(\\theta+\\delta)\\approx\\mathcal L(\\theta)+\\nabla_\\theta\\mathcal L^\\top\\delta$, maximized over unit $\\delta$ by $\\delta\\propto\\nabla_\\theta\\mathcal L$). Gradient descent steps the opposite way:\n$$ \\boxed{\\ \\theta\\leftarrow\\theta-\\eta\\,\\nabla_\\theta\\mathcal L\\ } $$with step size $\\eta\\gt 0$ ([CONFIG], Lesson 8). This update is \u0026ldquo;training writes to $\\theta$.\u0026rdquo; It touches every [LEARNED] quantity (Lessons 2–5) and nothing else: $\\eta$, $N$, $V$ etc. are constants here.\n$\\mathcal L$ is highly nonconvex in $\\theta$, so this finds a local descent path, not a global optimum — yet in the very high-dimensional regime of LLMs, local minima are empirically good and largely equivalent in loss. Convergence guarantees from convex optimization do not apply; what we have is an enormously effective heuristic.\nThe update rule is trivial to write down. The hard part is the term inside it: obtaining $\\nabla_\\theta\\mathcal L$ over all $P$ parameters at a cost we can afford. That is what the next section solves.\n2. The cost problem and reverse-mode autodiff A finite-difference estimate of one partial derivative, $\\partial\\mathcal L/\\partial\\theta_j\\approx[\\mathcal L(\\theta+he_j)-\\mathcal L(\\theta)]/h$, costs one forward pass; doing this for all $P$ parameters costs $P$ forward passes — utterly infeasible at $P\\sim10^{11}$.\nBackpropagation = reverse-mode automatic differentiation. Its defining property: it computes the gradient w.r.t. all $P$ parameters in a single backward sweep whose cost is $\\Theta(1)$ forward passes (typically ~2×). The asymmetry — cheap when there are many inputs and one scalar output — is exactly the LLM situation ($P$ inputs, one $\\mathcal L$), which is why reverse mode (not forward mode) is universal in deep learning.\nVector-Jacobian products View $f$ as a composition of primitive maps $\\mathbf a^{(\\ell)}=g_\\ell(\\mathbf a^{(\\ell-1)};\\theta_\\ell)$. Define the adjoint (the gradient of the scalar loss w.r.t. a given activation) $\\bar{\\mathbf a}^{(\\ell)}:=\\partial\\mathcal L/\\partial\\mathbf a^{(\\ell)}$. The chain rule, in the form actually implemented, propagates adjoints backward via vector-Jacobian products (VJPs):\n$$ \\bar{\\mathbf a}^{(\\ell-1)}=\\Big(\\frac{\\partial g_\\ell}{\\partial \\mathbf a^{(\\ell-1)}}\\Big)^{\\!\\top}\\bar{\\mathbf a}^{(\\ell)}, \\qquad \\bar\\theta_\\ell=\\Big(\\frac{\\partial g_\\ell}{\\partial \\theta_\\ell}\\Big)^{\\!\\top}\\bar{\\mathbf a}^{(\\ell)}. $$Crucially, no full Jacobian is ever materialized — each primitive supplies a function that maps an incoming adjoint to outgoing adjoints. That is why memory and compute stay linear in the size of the computation graph.\nThe algorithm Forward: compute and cache activations $\\mathbf a^{(\\ell)}$ up to scalar $\\mathcal L$. Seed: $\\bar{\\mathbf z}=\\hat{\\mathbf p}-\\mathbf y$ (Lesson 5). Backward: for $\\ell=N,\\dots,1$, apply the two VJPs above; accumulate parameter adjoints $\\bar\\theta_\\ell$. Update: feed $g=(\\bar\\theta_1,\\dots,\\bar\\theta_N)$ to the optimizer. The four steps are abstract until you watch them act on a single layer. The next section runs the two VJPs on concrete numbers, so the \u0026ldquo;apply the recipe and pass the adjoint down\u0026rdquo; loop becomes mechanical rather than symbolic.\nFigure 1. Why reverse mode, and not the obvious thing.\n3. Worked example: a linear layer For $\\mathbf u=W\\mathbf x$ (shapes $W\\in\\mathbb R^{m\\times n}$, $\\mathbf x\\in\\mathbb R^n$), given incoming adjoint $\\bar{\\mathbf u}=\\partial\\mathcal L/\\partial\\mathbf u$, the VJPs are\n$$ \\bar W=\\frac{\\partial\\mathcal L}{\\partial W}=\\bar{\\mathbf u}\\,\\mathbf x^\\top\\in\\mathbb R^{m\\times n} \\qquad\\text{(outer product → parameter gradient),} $$ $$ \\bar{\\mathbf x}=\\frac{\\partial\\mathcal L}{\\partial\\mathbf x}=W^\\top\\bar{\\mathbf u}\\in\\mathbb R^{n} \\qquad\\text{(matrix-vector → adjoint passed downstream).} $$Both reuse the cached forward activation $\\mathbf x$. The entire backward pass is millions of such local matrix multiplies, composed automatically. For batched inputs $X\\in\\mathbb R^{B\\times n}$ the parameter gradient sums over the batch, $\\bar W=\\bar U^\\top X$ — the origin of the batch-averaging in Lesson 6.\nWith numbers. Take\n$$ W=\\begin{pmatrix}1\u00262\\\\0\u00263\\end{pmatrix},\\quad \\mathbf x=\\begin{pmatrix}1\\\\1\\end{pmatrix} \\ \\Rightarrow\\ \\mathbf u=W\\mathbf x=\\begin{pmatrix}1\\cdot1+2\\cdot1\\\\0\\cdot1+3\\cdot1\\end{pmatrix}=\\begin{pmatrix}3\\\\3\\end{pmatrix}. $$Suppose the backward pass arrives at this layer with incoming adjoint $\\bar{\\mathbf u}=(1,\\,-2)^\\top$ (loss is rising in $u_1$, falling twice as fast in $u_2$). The two VJPs give, mechanically:\n$$ \\bar W=\\bar{\\mathbf u}\\,\\mathbf x^\\top=\\begin{pmatrix}1\\\\-2\\end{pmatrix}\\!\\begin{pmatrix}1\u00261\\end{pmatrix}=\\begin{pmatrix}1\u00261\\\\-2\u0026-2\\end{pmatrix}, \\qquad \\bar{\\mathbf x}=W^\\top\\bar{\\mathbf u}=\\begin{pmatrix}1\u00260\\\\2\u00263\\end{pmatrix}\\!\\begin{pmatrix}1\\\\-2\\end{pmatrix}=\\begin{pmatrix}1\\\\-4\\end{pmatrix}. $$Read off what each means. $\\bar W$ is the gradient written into this layer\u0026rsquo;s weights — the optimizer (Lesson 8) will nudge $W$ against it. $\\bar{\\mathbf x}$ is the adjoint handed to the layer below, which becomes its incoming $\\bar{\\mathbf u}$, and the same two-line recipe repeats all the way down to the embeddings. Sanity check on one entry: $\\partial\\mathcal L/\\partial W_{11}=\\bar u_1\\,x_1=1\\cdot1=1$, matching the top-left of $\\bar W$.\nSoftmax+CE seed, revisited. Differentiating softmax and cross-entropy separately would route a $1/\\hat p$ factor against the softmax Jacobian; fusing them gives the bounded $\\hat{\\mathbf p}-\\mathbf y$ directly. This is both a numerical-stability and a speed win, and it is why frameworks expose a fused cross_entropy_with_logits.\nOne layer behaves; a stack of a hundred need not. Because backprop multiplies these local maps on the way down, whether the gradient survives the trip depends on the spectrum of that product — which is exactly where the architectural choices of Lessons 3–4 were quietly buying us trainability.\nFigure 2. The two vector-Jacobian products, on the lesson\u0026rsquo;s numbers.\n4. Why the architecture choices from earlier lessons matter here Backprop multiplies many local Jacobians; their product\u0026rsquo;s spectrum governs trainability.\nVanishing/exploding gradients. Through a plain depth-$N$ stack, $\\bar{\\mathbf a}^{(0)}=\\big(\\prod_\\ell J_\\ell^\\top\\big)\\bar{\\mathbf a}^{(N)}$; if the $J_\\ell$ have spectral norm consistently $\\lt 1$ the product →0 (vanishing), if $\\gt 1$ it diverges (exploding). Residuals (Lesson 4) replace $J_\\ell$ with $I+J'_\\ell$, whose product retains an identity term, keeping gradient norms $O(1)$ across depth. Saturation. Where softmax (attention or output) saturates, its Jacobian $\\mathrm{diag}(\\mathbf a)-\\mathbf a\\mathbf a^\\top\\to0$ kills gradients — the reason for the $\\sqrt{d_k}$ scaling (Lesson 3). Normalization (Lesson 4) bounds activation scale, keeping the $J_\\ell$ well-conditioned. These are not aesthetic choices; they are what make $\\nabla_\\theta\\mathcal L$ informative enough to descend.\nA well-conditioned gradient is necessary but not sufficient: we also have to be able to hold the backward pass in memory. That cost, and the trick that contains it, is the next concern.\nFigure 3. What the product of local Jacobians does to the gradient.\n5. The memory cost of backprop, and checkpointing Reverse mode must keep forward activations alive until their adjoint is computed, so activation memory scales with (depth × batch × sequence length) — often exceeding parameter memory during training. Gradient (activation) checkpointing trades compute for memory: store activations only at a subset of layers and recompute the rest during the backward pass, cutting activation memory from $O(N)$ to $O(\\sqrt N)$ at the cost of one extra partial forward pass. This, plus mixed precision (Lesson 8), is what makes training large models fit on real hardware. These caches are pure [STATE] — discarded after each step.\nWe have so far reasoned as if $\\nabla_\\theta\\mathcal L$ were the gradient of the loss over the entire corpus. It never is — we cannot afford it, and as it turns out we do not want it. The final section makes the gradient we actually use precise.\n6. Stochastic gradient descent We never compute $\\nabla_\\theta\\mathcal L$ over the whole corpus. We estimate it on a random mini-batch:\n$$ g_B=\\frac1{|B|}\\sum_{i\\in B}\\nabla_\\theta\\ell_i,\\qquad \\mathbb E[g_B]=\\nabla_\\theta\\mathcal L\\ \\text{(unbiased)},\\quad \\mathrm{Var}[g_B]\\propto\\tfrac1{|B|}. $$So mini-batch gradients are unbiased estimators with variance shrinking as $1/|B|$. The noise is not merely tolerated — it has a mild regularizing effect, helping escape saddle points (abundant in high dimensions) and sharp minima. Batch size $|B|$ trades estimator variance against per-step cost (Lesson 8).\nSeparation of concerns. Backprop computes $g$ (the direction); it never modifies $\\theta$. The optimizer\u0026rsquo;s update rule modifies $\\theta$. Keep them distinct: differentiation ≠ optimization.\nLearned / config / state Quantity Category All $\\theta$ (target of the update) [LEARNED] $\\eta$, batch size, checkpointing policy [CONFIG] Cached activations, adjoints $\\bar{\\mathbf a}$, gradient $g$ [STATE] Conclusion Training is gradient descent $\\theta\\leftarrow\\theta-\\eta\\nabla_\\theta\\mathcal L$ on a nonconvex landscape; this update is the entirety of \u0026ldquo;training writes to $\\theta$.\u0026rdquo; The gradient over all $P$ parameters is computed in one backward sweep by reverse-mode autodiff, which propagates adjoints via vector-Jacobian products ($\\bar W=\\bar{\\mathbf u}\\mathbf x^\\top$, $\\bar{\\mathbf x}=W^\\top\\bar{\\mathbf u}$ for a linear layer) without ever forming a full Jacobian — cheap precisely because there are many parameters and one scalar loss. Residuals, the $\\sqrt{d_k}$ scaling, and normalization exist to keep the product of local Jacobians well-conditioned so gradients neither vanish nor explode. Activation checkpointing tames backprop\u0026rsquo;s memory. We descend on unbiased mini-batch estimates whose $1/|B|$-variance noise aids generalization. Backprop produces the direction; the optimizer takes the step.\nNext: the optimizers, schedules, and regularizers that turn this bare update into a stable large-scale training recipe.\n← Lesson 6 — The Loss Function · Course index · Next: Lesson 8 — Optimizers, Learning Rate \u0026amp; Batches → Appendices: A — Acronyms · B — Formulas · C — Attention Variants The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/tutorials/llm-lessons/lesson-07-backpropagation-and-gradient-descent/","summary":"\u003ch1 id=\"lesson-7--backpropagation--gradient-descent\"\u003eLesson 7 — Backpropagation \u0026amp; Gradient Descent\u003c/h1\u003e\n\u003cp\u003e\u003cem\u003eTechnical course on LLM mechanics — Lesson 7 of 10.\u003c/em\u003e\u003c/p\u003e\n\u003chr\u003e\n\u003ch2 id=\"recap-of-lesson-6\"\u003eRecap of Lesson 6\u003c/h2\u003e\n\u003cp\u003eCross-entropy $=H(p)+D_{\\mathrm{KL}}(p\\|\\hat p)$, so minimizing it minimizes KL to the true distribution, floored by language\u0026rsquo;s entropy $H(p)$. With one-hot targets it is $-\\log\\hat p_{x_t}$, a proper scoring rule (hence calibration, hence why we don\u0026rsquo;t train on accuracy). Perplexity $e^{\\mathcal L}$ is the effective branching factor. The logit gradient is $\\partial\\mathcal L/\\partial\\mathbf z=\\hat{\\mathbf p}-\\mathbf y$. This lesson turns that single gradient into $\\nabla_\\theta\\mathcal L\\in\\mathbb R^P$ and specifies the update — the precise mechanics of \u003cem\u003ewhat gets written\u003c/em\u003e.\u003c/p\u003e","title":"Backpropagation and Gradient Descent"},{"content":"Lesson 7 — Distributed Training: Expert Parallelism Technical course on MoE LLM mechanics — Lesson 7 of 10.\nRecap of Lesson 6 To run experts efficiently on parallel hardware, each is given a fixed capacity $C=\\lceil C_f kT/E\\rceil$; overflow tokens are dropped onto the residual path, and the batch is executed via a dispatch tensor (scatter tokens into fixed-size expert buffers) and a gate-weighted combine tensor (un-scatter the outputs). That formulation assumed every expert lived on one device. They do not. This lesson distributes the experts across many accelerators, turning dispatch and combine into the two all-to-all network collectives that make MoE training distinctively communication-bound.\nThe memory side of Lesson 1\u0026rsquo;s asymmetry now comes due. An MoE holds far more parameters than it computes, and those parameters — chiefly the experts — must physically live somewhere. They do not fit on one device, and replicating them everywhere would waste exactly the memory MoE was supposed to spend wisely. The answer is to shard the experts, and sharding the experts means tokens must travel to reach them.\n1. The parallelism menu, and where MoE adds a row Large dense models are already trained with several axes of parallelism, all of which MoE inherits unchanged:\nData parallelism: replicate the full model on each device, split the batch; average gradients. Communication: all-reduce of gradients. Tensor parallelism: split individual weight matrices across devices within a layer; communication: all-reduce of partial activations. Pipeline parallelism: put different layers on different devices; communication: point-to-point activations between stages. MoE adds a new axis specific to the expert layer:\nExpert parallelism: place different experts on different devices. Each device holds a subset of the $E$ experts for a given MoE layer. Communication: all-to-all, twice per MoE layer. Expert parallelism is what makes the large $P_{\\text{tot}}$ feasible: with $E$ experts spread over $G$ devices, each device stores only $E/G$ experts, so the per-device expert memory drops by $G$. This is the direct mechanism by which \u0026ldquo;expensive bytes\u0026rdquo; are made affordable — you buy more devices to hold more experts, and each device\u0026rsquo;s share stays bounded.\nFigure 1. Four axes of parallelism, three of them inherited.\n2. Why all-to-all, and what it costs Here is the geometry of the problem. Tokens are distributed across devices (by data parallelism — each device processes its slice of the batch), but the expert a token needs may live on a different device. So before the experts can run, every device must send each of its tokens to whichever device holds that token\u0026rsquo;s selected expert — and after the experts run, every device must send the results back to where the tokens came from. Each device thus exchanges data with every other device: an all-to-all collective.\nTwo all-to-all operations bracket every MoE layer:\nflowchart LR subgraph step1[\"per MoE layer\"] direction LR tok[\"tokens on eachdevice (by data-parallel split)\"] --\u003e|\"all-to-all #1(dispatch)\"| exp[\"tokens regroupedby expert/device\"] exp --\u003e|\"local expert FFNs\"| out[\"expert outputson expert device\"] out --\u003e|\"all-to-all #2(combine)\"| back[\"outputs returnedto token's home device\"] end Dispatch all-to-all: route each token to the device holding its selected expert(s). (local) run the experts now resident on this device, Combine all-to-all: send each expert output back to the device that owns the token. The all-to-all is the realization of Lesson 6\u0026rsquo;s dispatch/combine tensors across the network. It moves $O(T\\cdot d)$ activation data twice per MoE layer, over every device pair, on every forward pass (and again, transposed, on the backward pass). This communication is the defining cost of MoE training: where a dense model\u0026rsquo;s inter-device traffic is gradient all-reduce (once per step, overlappable with compute) and pipeline hand-offs, an MoE adds two all-to-alls per MoE layer per step that sit on the critical path of the forward and backward passes. At scale, MoE training is frequently communication-bound, and a large fraction of engineering effort goes into overlapping these all-to-alls with expert compute and into topology-aware placement.\nFigure 2. Two all-to-all collectives bracket every MoE layer.\n3. Why balance is now also a systems requirement Lesson 5 motivated load balancing for model quality. Expert parallelism gives it a second, equally hard edge. Because each device runs the experts it holds, the device with the most-loaded expert determines the layer\u0026rsquo;s wall-clock time — every other device waits at the all-to-all barrier for the slowest one. An imbalanced router therefore creates stragglers: idle accelerators stalled behind one overloaded expert.\nThis is exactly why the fixed capacity of Lesson 6 exists in the distributed setting: a fixed $C$ caps the work (and the message size) per expert, bounding the straggler effect and keeping the all-to-all messages a fixed, schedulable shape — at the price of dropping overflow. Balance (Lesson 5), capacity (Lesson 6), and communication (this lesson) are thus three views of one constraint: a sparse model is only as fast as its busiest expert, so the router must spread load, the buffers must be bounded, and the network must move tokens to wherever the work was sent.\nContrast with the dense model. A dense model\u0026rsquo;s computation is local and uniform: every device does identical work on its shard, and the only required communication is gradient all-reduce, which can be overlapped with the backward pass. An MoE inserts a data-dependent shuffle (two all-to-alls per MoE layer) into the critical path, whose cost and balance depend on the runtime routing pattern, not just the model shape. Dense parallelism is a static partition of fixed work; MoE parallelism must additionally move variable amounts of data to follow the router\u0026rsquo;s per-token decisions. This is the second great divergence of MoE from dense training (the first being the balancing loss of Lesson 5), and the two are intertwined: balancing is what keeps the communication efficient.\nFigure 3. A sparse model is only as fast as its busiest expert.\n4. GShard and the standard recipe GShard (Google) is the work that made expert parallelism practical at scale and fixed the now-standard pattern: shard experts across devices, use a capacity factor with token dropping (Lesson 6), add the auxiliary balancing loss (Lesson 5), and implement dispatch/combine as all-to-all collectives. The combination — expert sharding + capacity + auxiliary loss + all-to-all — is the canonical MoE training stack that Switch, GLaM, Mixtral, and DeepSeek-MoE all descend from, varying the details (granularity, $k$, balancing scheme) but keeping the skeleton. Expert parallelism is usually composed with the other axes (data + tensor + pipeline + expert), giving the multi-dimensional device meshes used to train frontier MoE models.\n5. Learned / config / state Quantity Category Expert and router weights (sharded across devices) [LEARNED] Expert-parallel degree $G$, device mesh layout, capacity factor $C_f$, all-to-all schedule [CONFIG] Dispatched token buffers in flight, all-to-all messages, per-device expert load [STATE] Conclusion An MoE holds more parameters than fit on one device, so the experts are sharded across accelerators — expert parallelism, a new axis composed with data/tensor/pipeline parallelism that cuts per-device expert memory by the shard count $G$. The price is communication: because a token\u0026rsquo;s selected expert may live on another device, every MoE layer is bracketed by two all-to-all collectives (dispatch tokens to expert devices, combine results back), on the critical path of both forward and backward passes. This makes MoE training distinctively communication-bound, and it gives load balancing a systems meaning on top of its quality meaning — the layer runs at the speed of its busiest expert, so balance (Lesson 5) and bounded capacity (Lesson 6) are what keep the all-to-all efficient. GShard\u0026rsquo;s shard + capacity + aux-loss + all-to-all recipe is the skeleton every modern MoE inherits. Versus a dense model — whose only required traffic is overlappable gradient all-reduce — the MoE inserts a data-dependent shuffle that follows the router\u0026rsquo;s decisions at runtime.\nWe have now built the full MoE training apparatus: experts, router, balancing, capacity, and communication. The next lesson steps back to the dynamics of running this apparatus — the instabilities it introduces, the specialization that emerges, and the scaling laws that justify the whole enterprise.\n← Lesson 6 — Capacity \u0026amp; Token Dropping · Course index · Next: Lesson 8 — Training Dynamics, Stability \u0026amp; Scaling Laws → Appendices: A — MoE Family · B — Dense vs. MoE · C — Acronyms The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/tutorials/moe-lessons/lesson-07-expert-parallelism/","summary":"\u003ch1 id=\"lesson-7--distributed-training-expert-parallelism\"\u003eLesson 7 — Distributed Training: Expert Parallelism\u003c/h1\u003e\n\u003cp\u003e\u003cem\u003eTechnical course on MoE LLM mechanics — Lesson 7 of 10.\u003c/em\u003e\u003c/p\u003e\n\u003chr\u003e\n\u003ch2 id=\"recap-of-lesson-6\"\u003eRecap of Lesson 6\u003c/h2\u003e\n\u003cp\u003eTo run experts efficiently on parallel hardware, each is given a fixed \u003cstrong\u003ecapacity\u003c/strong\u003e $C=\\lceil C_f kT/E\\rceil$; overflow tokens are \u003cstrong\u003edropped\u003c/strong\u003e onto the residual path, and the batch is executed via a \u003cstrong\u003edispatch tensor\u003c/strong\u003e (scatter tokens into fixed-size expert buffers) and a gate-weighted \u003cstrong\u003ecombine tensor\u003c/strong\u003e (un-scatter the outputs). That formulation assumed every expert lived on one device. They do not. This lesson distributes the experts across many accelerators, turning dispatch and combine into the two \u003cstrong\u003eall-to-all\u003c/strong\u003e network collectives that make MoE training distinctively communication-bound.\u003c/p\u003e","title":"Distributed Training: Expert Parallelism"},{"content":"Lesson 7 — Evaluating an Encoder Technical course on encoder LLM mechanics — Lesson 7 of 10.\nRecap of Lesson 6 The encoder\u0026rsquo;s representations $H^{(N)}$ become predictions through heads: the MLM head (tied vocabulary projection, pretraining only), pooling + a sequence classifier, a per-token classifier, and a span head. The decoder routes every task through one generative head; the encoder swaps in typed heads. We now ask how to measure an encoder. The decoder has a clean intrinsic yardstick — perplexity — that an encoder cannot use, which forces a different evaluation philosophy: encoders are judged mostly by what they enable downstream.\n1. The pretraining loss, restated as a metric The training objective (Lesson 5) doubles as the first evaluation number. The MLM loss is the cross-entropy at masked positions,\n$$ \\mathcal L_{\\text{MLM}}=-\\,\\mathbb E\\sum_{i\\in\\mathcal M}\\log P_\\theta(x_i\\mid\\tilde x), $$measured in nats (natural log) or bits (log base 2), exactly as the decoder course defined cross-entropy. As $H(p)+D_{\\mathrm{KL}}(p\\Vert\\hat p)$ it is a proper scoring rule, minimized when the model\u0026rsquo;s masked-token distribution matches the true conditional. A lower held-out MLM loss means the model reconstructs masked tokens better. So far this mirrors the decoder.\nThe divergence is in what you can build on top of that loss.\n2. Why standard perplexity does not apply For a decoder, perplexity is the clean summary of held-out loss:\n$$ \\mathrm{PPL}=\\exp\\Big(-\\tfrac1n\\sum_{t=1}^{n}\\log P_\\theta(x_t\\mid x_{\\lt t})\\Big), $$the geometric-mean inverse probability the model assigns to each next token. It is well-defined because the chain rule makes $\\prod_t P_\\theta(x_t\\mid x_{\\lt t})$ a valid joint probability of the whole sequence — perplexity is a property of that joint distribution.\nAn MLM encoder has no such joint distribution. It defines a set of conditionals $P_\\theta(x_i\\mid\\tilde x)$, one per masked position given the rest, but these conditionals are not mutually consistent: there is no single joint $P_\\theta(x_1,\\dots,x_n)$ whose conditionals they are. (Each is conditioned on a different corrupted context, and they overlap in incompatible ways.) You therefore cannot multiply them into a sequence probability, and the decoder\u0026rsquo;s perplexity formula is simply not defined for an encoder. This is a direct consequence of dropping the autoregressive factorization in Lesson 1 — lose the chain rule and you lose perplexity with it.\nContrast with the decoder. Perplexity is the decoder\u0026rsquo;s headline intrinsic metric precisely because it is a coherent autoregressive joint model. The encoder, modeling only mutually-inconsistent masked conditionals, has no sequence probability to be perplexed by. Reporting \u0026ldquo;BERT\u0026rsquo;s perplexity\u0026rdquo; without qualification is a category error.\nFigure 1. Why an encoder has no perplexity.\n3. Pseudo-perplexity: the closest intrinsic substitute When an intrinsic number is genuinely needed (e.g. comparing two MLM checkpoints on the same data), the field uses pseudo-(log-)likelihood (PLL) and its exponential, pseudo-perplexity (PPPL). Mask each token one at a time, score its conditional given all the others, and sum:\n$$ \\mathrm{PLL}(x)=\\sum_{i=1}^{n}\\log P_\\theta\\big(x_i\\mid x_{\\setminus i}\\big),\\qquad \\mathrm{PPPL}=\\exp\\!\\Big(-\\tfrac1n\\,\\mathrm{PLL}(x)\\Big), $$where $x_{\\setminus i}$ is the sequence with position $i$ (alone) masked. This requires $n$ forward passes per sentence (one per masked position), versus the decoder\u0026rsquo;s single pass for true perplexity — a real cost difference. PPPL is a useful relative metric (lower is better, comparable across MLMs trained the same way) but it is not a true perplexity: it does not measure a coherent generative probability, and PPPL values are not comparable to a decoder\u0026rsquo;s PPL.\nTakeaway. Pseudo-perplexity is the encoder\u0026rsquo;s pragmatic stand-in for perplexity: defined, monotone in model quality, but $n\\times$ more expensive to compute and not interchangeable with a decoder\u0026rsquo;s perplexity number.\nFigure 2. The price of a pseudo-perplexity.\n4. The real yardstick: downstream benchmarks Because intrinsic metrics are weak for encoders, the field evaluates them extrinsically — fine-tune (Lesson 9) on a battery of understanding tasks and report task scores. This is the dominant, decisive evaluation for encoders.\nGLUE (General Language Understanding Evaluation) and the harder SuperGLUE aggregate ~8–10 tasks into a single number: single-sentence classification (CoLA grammaticality, SST-2 sentiment), sentence-pair similarity/paraphrase (MRPC, STS-B, QQP), and natural-language inference (MNLI, RTE, QNLI). SQuAD measures extractive QA via Exact Match and token-level F1. Token-labeling is measured on CoNLL-2003 NER (entity-level F1). Each task uses the appropriate head from Lesson 6.\nThe logic: an encoder\u0026rsquo;s job is to produce representations that transfer to understanding tasks, so the only evaluation that matters is how well those tasks do. A model with slightly worse MLM loss but better GLUE is the better encoder. This is why nearly every encoder paper (Appendix A) leads with a GLUE/SQuAD table, not a perplexity figure.\nflowchart LR M[\"pretrained encoder\"] --\u003e A[\"fine-tune + eval on GLUE(NLI, sentiment, paraphrase)\"] M --\u003e B[\"fine-tune + eval on SQuAD(extractive QA: EM / F1)\"] M --\u003e C[\"fine-tune + eval on CoNLL NER(entity F1)\"] A --\u003e R[\"report task scores =the real verdict\"] B --\u003e R C --\u003e R Contrast with the decoder. A modern decoder is increasingly evaluated zero-/few-shot — you prompt it and read its generated answer, no fine-tuning. An encoder is evaluated after fine-tuning a head on each task. The decoder shows capability through generation under instruction; the encoder shows capability through transfer of representations. Different machines, different rulers.\nFigure 3. What actually decides an encoder\u0026rsquo;s worth.\n5. Embedding-quality evaluation A growing fraction of encoder use is producing sentence/document embeddings for retrieval and similarity (Lesson 10). These are evaluated on their own benchmarks — semantic-textual-similarity correlation (STS), and the broad MTEB (Massive Text Embedding Benchmark) covering retrieval, clustering, reranking, and classification over many datasets. The metric is whether geometric relationships in embedding space (cosine similarity) track human judgments of meaning. This evaluation mode barely exists for decoders, whose embeddings are a side product rather than the deliverable.\nConclusion Evaluating an encoder is shaped by what it lacks: with no autoregressive joint distribution, the decoder\u0026rsquo;s perplexity is undefined, because the MLM model\u0026rsquo;s masked conditionals are mutually inconsistent and cannot be multiplied into a sequence probability. The held-out MLM loss (nats/bits) is a valid relative metric, and pseudo-perplexity — masking one token at a time over $n$ passes — is the closest intrinsic substitute, useful for comparing MLMs but not a true perplexity and not comparable to a decoder\u0026rsquo;s. The metrics that actually decide an encoder\u0026rsquo;s worth are extrinsic: fine-tune and report GLUE/SuperGLUE, SQuAD, CoNLL NER, and, for embedding models, STS/MTEB. Where decoders are increasingly judged zero-shot by their generations, encoders are judged by how well their representations transfer after fine-tuning.\nThat makes fine-tuning the linchpin of the whole encoder story. Lesson 8 first looks at how pretraining is actually run at scale, then Lesson 9 covers fine-tuning the four task heads.\n← Previous: Lesson 6 — From Representations to Predictions · Course index · Next: Lesson 8 — Pretraining Dynamics → Appendices: A — Encoder Family · B — Encoder vs. Decoder Cheat Sheet · C — Acronyms The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/tutorials/encoder-deep-dive/lesson-07-loss-and-evaluation/","summary":"\u003ch1 id=\"lesson-7--evaluating-an-encoder\"\u003eLesson 7 — Evaluating an Encoder\u003c/h1\u003e\n\u003cp\u003e\u003cem\u003eTechnical course on encoder LLM mechanics — Lesson 7 of 10.\u003c/em\u003e\u003c/p\u003e\n\u003chr\u003e\n\u003ch2 id=\"recap-of-lesson-6\"\u003eRecap of Lesson 6\u003c/h2\u003e\n\u003cp\u003eThe encoder\u0026rsquo;s representations $H^{(N)}$ become predictions through \u003cstrong\u003eheads\u003c/strong\u003e: the MLM head (tied vocabulary projection, pretraining only), pooling + a sequence classifier, a per-token classifier, and a span head. The decoder routes every task through one generative head; the encoder swaps in typed heads. We now ask how to \u003cem\u003emeasure\u003c/em\u003e an encoder. The decoder has a clean intrinsic yardstick — perplexity — that an encoder cannot use, which forces a different evaluation philosophy: encoders are judged mostly by what they \u003cem\u003eenable downstream\u003c/em\u003e.\u003c/p\u003e","title":"Loss and Evaluation"},{"content":"Lesson 7 — Pretraining Objectives: Span Corruption vs. Denoising Technical course on encoder–decoder transformers — Lesson 7 of 10.\nRecap of Lesson 6 The encoder–decoder is one point on a masking spectrum: a prefix-LM folds the two stacks into one with a block mask (bidirectional prefix, causal suffix), the decoder-only sits at the fully-causal end, and encoder-only at the fully-bidirectional end. All differ only in mask and in whether $K,V$ come from a second sequence. T5\u0026rsquo;s ablation found the two-stack design strongest for seq2seq transfer. We now enter the training phase and ask: on what objective is a fresh encoder–decoder pretrained? The answer — denoising — is where T5 and BART diverge most from both MLM and next-token prediction.\n1. The problem: how do you self-supervise two stacks? A decoder-only model has an obvious free objective: next-token prediction, where the label at position $t$ is simply $x_{t+1}$. An encoder-only model (BERT) has masked-language-modeling: corrupt 15% of tokens, predict them in place. But an encoder–decoder needs an objective that exercises both the bidirectional reader and the autoregressive writer at once, with no human labels — because the whole point of pretraining is to use raw text at $10^{12}$ scale. The unifying idea both T5 and BART adopt is denoising: corrupt the input text in some way, feed the corrupted version to the encoder, and train the decoder to reconstruct the missing/original content autoregressively. The corruption is what creates a (input, output) pair from a single unlabeled document.\nThe denoising recipe. Encoder reads a corrupted document $\\tilde x$; decoder generates the clean (or missing) content $y$. The loss is ordinary cross-entropy on the decoder\u0026rsquo;s targets — the architecture and loss are unchanged from Lesson 5; only the way we construct $(\\tilde x, y)$ from raw text is new.\nThis single recipe, instantiated two different ways, is T5 and BART.\nFigure 1. The denoising recipe: one document becomes a supervised pair.\n2. T5: span corruption with sentinel tokens T5\u0026rsquo;s instantiation is span corruption. Take a document, randomly select spans of tokens totaling a fraction (T5 uses ~15%) of the text, with an average span length of ~3 tokens. Replace each contiguous span with a single unique sentinel token \u0026lt;X\u0026gt;, \u0026lt;Y\u0026gt;, \u0026lt;Z\u0026gt;, … The encoder sees the document with spans collapsed to sentinels; the decoder is trained to emit, for each sentinel, the sentinel followed by the tokens it stood for, ending with a final sentinel.\nConcretely, for the original \u0026ldquo;Thank you for inviting me to your party last week\u0026rdquo;:\n$$ \\begin{aligned} \\textbf{Encoder input } \\tilde x:\\quad \u0026 \\text{Thank you } \\texttt{\\textless{}X\\textgreater{}} \\text{ me to your party } \\texttt{\\textless{}Y\\textgreater{}} \\text{ week}\\\\ \\textbf{Decoder target } y:\\quad \u0026 \\texttt{\\textless{}X\\textgreater{}} \\text{ for inviting } \\texttt{\\textless{}Y\\textgreater{}} \\text{ last } \\texttt{\\textless{}Z\\textgreater{}} \\end{aligned} $$(Here the span \u0026ldquo;for inviting\u0026rdquo; became \u0026lt;X\u0026gt; and \u0026ldquo;last\u0026rdquo; became \u0026lt;Y\u0026gt;; \u0026lt;Z\u0026gt; marks the end.) Two design payoffs make this clever:\nThe target is short. The decoder only generates the corrupted tokens plus sentinels, not the whole document — so the sequence the decoder produces is a small fraction of the input length, which makes pretraining cheap relative to reconstructing everything. It exercises both stacks correctly. The encoder must build bidirectional representations rich enough to infer the missing spans from both sides (the word after a sentinel matters as much as the word before — exactly the bidirectional advantage of Lesson 2), and the decoder must generate them in order, conditioned via cross-attention. The sentinels are added to the vocabulary as ordinary [LEARNED] embeddings; the corruption rate (15%) and mean span length (3) are [CONFIG]; which spans get corrupted on a given example is sampled, i.e. [STATE] for that batch.\nFigure 2. Span corruption, on the lesson\u0026rsquo;s own sentence.\n3. BART: five flavors of noise BART generalizes denoising into a family of corruptions, and notably allows ones that change the sequence length — which only an encoder–decoder (with its decoupled input/output lengths, Lesson 1) can cleanly handle. BART\u0026rsquo;s decoder always reconstructs the entire original document, which is more expensive than T5\u0026rsquo;s span-only target but makes the model a true denoising autoencoder. The five noises:\nToken masking — replace random tokens with \u0026lt;mask\u0026gt; (BERT-style), but predict the whole document, not just the masked slots. Token deletion — delete random tokens; the model must figure out where something is missing as well as what. Text infilling — replace spans (sampled lengths, including length 0) with a single \u0026lt;mask\u0026gt; each — close to T5\u0026rsquo;s span idea, and BART found this the most important noise. Sentence permutation — shuffle the order of sentences; the decoder must restore the original order. Document rotation — rotate the document to start at a random token; the model must find the true start. Infilling and sentence permutation, used together, gave BART its best downstream results. The variety is the point: by forcing the model to undo many kinds of damage, the encoder learns robust representations and the decoder learns fluent generation — which is exactly the pair of skills summarization and translation need.\nflowchart LR doc[\"clean document\"] --\u003e noise[\"apply noise(span / mask / delete / permute / rotate)\"] noise --\u003e enc[\"Encoder reads x̃ (corrupted)\"] enc -.H_enc.-\u003e dec[\"Decoder reconstructs y (clean/missing)\"] dec --\u003e loss[\"cross-entropy on y\"] Figure 3. One noise, or five.\n4. The loss is the same CE you already know Whatever the corruption, the training loss is the ordinary teacher-forced cross-entropy over the decoder\u0026rsquo;s target positions (Lesson 8 details teacher forcing):\n$$ \\mathcal L(\\theta)=-\\frac{1}{|y|}\\sum_{t=1}^{|y|}\\log P_\\theta\\big(y_t\\mid y_{\\lt t},\\,\\tilde x\\big) =-\\frac{1}{|y|}\\sum_{t}\\log \\hat p_{\\,t,\\,y_t}, $$and the logit gradient is the familiar $\\partial\\mathcal L/\\partial\\mathbf z_t=\\hat{\\mathbf p}_t-\\mathbf y_t$ (predicted minus one-hot target), which seeds backprop through the decoder, across the cross-attention bridge, and into the encoder (Lesson 4 §5). Nothing about the optimization is special to encoder–decoders; AdamW, warmup-cosine schedules, and gradient clipping carry over unchanged. The objective lives entirely in how $(\\tilde x, y)$ is built.\n5. Contrast with the other two families It helps to line all three self-supervised objectives up, because they are often conflated.\nFamily Objective What predicts what Generates? Encoder-only (BERT) MLM: mask ~15% in place bidirectional encoder predicts masked tokens at their positions no — fills blanks, no autoregression Decoder-only (GPT) Next-token CE causal stack predicts $x_{t+1}$ from $x_{\\le t}$ yes, but no separate input encoding Encoder–decoder (T5/BART) Denoising / span corruption encoder reads corrupted $\\tilde x$; decoder autoregressively generates clean content yes, conditioned on bidirectional encoding The encoder–decoder objective is, in a sense, MLM and next-token prediction fused: like MLM it corrupts and reconstructs and rewards bidirectional reading; like next-token prediction it generates autoregressively. That fusion is only possible because the architecture has both a bidirectional reader and an autoregressive writer joined by cross-attention. The objective and the architecture are made for each other — which is why \u0026ldquo;span corruption on a decoder-only model\u0026rdquo; or \u0026ldquo;next-token prediction on BERT\u0026rdquo; are not natural things to do, while denoising on an encoder–decoder is.\nLearned / config / state Quantity Category Sentinel/\u0026lt;mask\u0026gt; token embeddings, all model weights [LEARNED] Corruption type, rate (~15%), mean span length, noise mix [CONFIG] Which tokens/spans are corrupted on a given example; the resulting $(\\tilde x,y)$ [STATE] Logits, probabilities, attention matrices [STATE] Conclusion Encoder–decoders are pretrained by denoising: corrupt a raw document into $\\tilde x$, feed it to the bidirectional encoder, and train the autoregressive decoder to reconstruct the clean or missing content $y$ — turning unlabeled text into supervised pairs with no human annotation. T5 uses span corruption (contiguous spans → single sentinel tokens; the decoder emits only the missing spans, keeping targets short), while BART mixes five noises (masking, deletion, infilling, sentence permutation, document rotation) and reconstructs the whole document. Either way the loss is the ordinary teacher-forced cross-entropy with the familiar $\\hat{\\mathbf p}-\\mathbf y$ logit gradient — the objective\u0026rsquo;s novelty is purely in how the (corrupted input, clean output) pair is built. The recipe fuses MLM\u0026rsquo;s bidirectional reconstruction with next-token generation, which is precisely the pair of stacks the architecture provides.\nWe have the objective. Lesson 8 turns it into a training procedure: teacher forcing, the shifted decoder inputs, training all output positions in parallel, the gradient\u0026rsquo;s path through cross-attention, and the exposure-bias gap that sets up inference.\n← Lesson 6 — Prefix-LM · Course index · Next: Lesson 8 — Training: Teacher Forcing → Appendices: A — Formulas · B — Architecture Comparison · C — Acronyms The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/tutorials/encoder-decoder-lessons/lesson-07-pretraining-objectives/","summary":"\u003ch1 id=\"lesson-7--pretraining-objectives-span-corruption-vs-denoising\"\u003eLesson 7 — Pretraining Objectives: Span Corruption vs. Denoising\u003c/h1\u003e\n\u003cp\u003e\u003cem\u003eTechnical course on encoder–decoder transformers — Lesson 7 of 10.\u003c/em\u003e\u003c/p\u003e\n\u003chr\u003e\n\u003ch2 id=\"recap-of-lesson-6\"\u003eRecap of Lesson 6\u003c/h2\u003e\n\u003cp\u003eThe encoder–decoder is one point on a masking spectrum: a \u003cstrong\u003eprefix-LM\u003c/strong\u003e folds the two stacks into one with a block mask (bidirectional prefix, causal suffix), the decoder-only sits at the fully-causal end, and encoder-only at the fully-bidirectional end. All differ only in mask and in whether $K,V$ come from a second sequence. T5\u0026rsquo;s ablation found the two-stack design strongest for seq2seq transfer. We now enter the \u003cstrong\u003etraining\u003c/strong\u003e phase and ask: on what objective is a fresh encoder–decoder pretrained? The answer — \u003cem\u003edenoising\u003c/em\u003e — is where T5 and BART diverge most from both MLM and next-token prediction.\u003c/p\u003e","title":"Pretraining Objectives: Span Corruption vs. Denoising"},{"content":"Chapter 7 — The Committee: Mixture-of-Experts Every model we have met so far shares a hidden assumption: when it processes a word, it uses all of itself to do it. Every one of its billions of knobs participates in every word. This is so natural that it is easy to miss as an assumption at all — but it has an expensive consequence, and the family of models in this chapter exists to escape it. These are mixture-of-experts models, the family that includes Switch, Mixtral, and DeepSeek, and their central idea is to build a model that is vast in what it knows while only switching on a small part of itself for any given word.\nThe weld between size and cost Start with the problem. In an ordinary model — called a dense model, because all of it is always engaged — capacity and cost are welded together. The only way to make such a model know more is to make it bigger, and making it bigger makes it proportionally slower and more expensive, because every newly added part now has to fire for every single word. Picture a vast factory where, to process one item, you must switch on every machine on the floor, including the ones with nothing to do with that item. Want more capability? Add more machines — but now they all run, all the time, for everything. There is no way to grow the factory\u0026rsquo;s abilities without growing its electricity bill in lockstep.\nA mixture-of-experts model breaks that weld. Instead of one monolithic machine that handles everything, it keeps a whole bank of smaller specialists — called experts — and adds a small decision-maker — the router — that looks at each incoming word and sends it to just a couple of the most relevant experts. The other experts stay switched off for that word. The model can therefore hold an enormous amount of knowledge across all its experts while only doing a small amount of work on any given word.\nThe hospital One analogy carries this entire chapter, so it is worth setting up carefully. Picture a large teaching hospital with hundreds of specialist doctors on staff. A patient — think of each word as a patient — arrives and is met by a triage nurse, who takes one look and sends them to the two specialists best suited to their case. The hospital\u0026rsquo;s total expertise is enormous, because every specialty is represented somewhere on staff. But no single patient consumes the whole hospital. They see two doctors and go home. The triage nurse is the router; the doctors are the experts; the patient is a word passing through. That is the whole idea. Everything else in this chapter is the practical machinery a hospital like that needs in order to actually function.\nTwo different sizes that everyone confuses Once a model only uses part of itself, the single notion of \u0026ldquo;model size\u0026rdquo; splits into two numbers, and keeping them apart is essential. The first is the total number of parameters — everything the model stores, every expert whether or not it is used on a given word. This is the model\u0026rsquo;s capacity, the full breadth of what it can know. It is the size of the hospital\u0026rsquo;s entire staff. The second is the number of activated parameters — the much smaller slice actually put to work on any one word, namely the router plus the two-or-so experts it picked. This is what determines speed and running cost. It is the handful of doctors a single patient actually sees.\nConcrete figures make the gap vivid. Mixtral stores around 47 billion parameters but only puts about 13 billion of them to work on each word. DeepSeek\u0026rsquo;s large model stores 671 billion but works with only about 37 billion at a time. The whole art of designing one of these models is buying a large total — more knowledge, better answers — while keeping the activated amount, and therefore the per-word cost, small. You get the staff of a giant hospital but pay, per patient, for a small clinic. A useful slogan to carry forward: cheap to compute, expensive to store. That asymmetry will reappear at every stage.\nFigure 1. Two sizes: what a committee stores, and what one word uses.\nWhat an expert is, and what the router does There is nothing exotic inside an expert. Recall from Chapter 4 that each block of a model has a stage where every word is processed on its own, after attention has gathered its context — and that this stage is where most of the model\u0026rsquo;s stored knowledge lives. A mixture-of-experts model simply makes many copies of that stage — eight of them, or sixty-four, or more — each with its own independently learned settings, and calls each copy an expert. They all have the same shape but different \u0026ldquo;training,\u0026rdquo; like a roomful of doctors who attended the same medical school and then each specialized differently. The attention machinery is left completely untouched. The whole mixture-of-experts story lives in this one swapped-out stage.\nThe router is astonishingly small — a tiny sliver of the model — yet it controls everything. For each word, it produces a score for every expert, essentially answering \u0026ldquo;how much does this word want this expert?\u0026rdquo;, then picks the few highest-scoring ones and assigns each a weight reflecting its confidence. Only those chosen experts run; their outputs are blended together according to the router\u0026rsquo;s weights; the rest of the experts contribute nothing because they were never consulted. The triage nurse glances at the patient, picks the two best specialists, and sends everyone else back to their offices.\nSome newer designs add refinements. Shared experts are a few that every word passes through no matter what, handling the common everyday processing all words need — the general intake exam everyone gets before being routed to specialists — so the chosen specialists don\u0026rsquo;t each have to relearn the basics. Fine-grained experts slice each expert thinner and use many more of them, giving the router far more combinations to choose from and room to specialize more precisely.\nFigure 2. The triage nurse, and the two doctors the patient sees.\nThe deep difficulty: learning from a choice, and the lazy router Here lies the central headache of the whole approach, and it is worth understanding because it explains most of what makes these models tricky. Recall from Chapter 1 that a model learns by nudging its knobs in whatever direction reduces its mistakes. But the router\u0026rsquo;s action is a choice — send this word to expert three, not expert seven — and a choice is all-or-nothing. You cannot send a word slightly to expert three. So the usual smooth learning signal has nothing to grab onto at the moment of selection. The router can be told \u0026ldquo;the experts you picked turned out well, lean into them\u0026rdquo; or \u0026ldquo;they turned out badly, trust them less,\u0026rdquo; but it never gets a clean signal saying \u0026ldquo;you should have picked a completely different expert,\u0026rdquo; because the expert it didn\u0026rsquo;t pick produced no result to compare against. The router learns to weight its winners well; it gets no clean nudge to change its winners.\nThis blind spot breeds a nasty failure called expert collapse. Suppose, just by luck at the start, a few experts are slightly better than the rest. The router sends them more words. More words means more practice, so they improve further. So the router sends them yet more. Meanwhile the neglected experts get almost no words, never improve, and fall further out of favor. The end state is a handful of experts doing nearly all the work while the rest sit idle as dead weight. This is disastrous, because capacity was the entire point: if you built a sixty-four-expert model but the router only ever uses four of them, you paid the storage bill for sixty-four and got the brains of four. In the hospital, the triage nurse keeps funneling every patient to the same two overworked doctors while the rest read magazines — you have staffed a huge hospital and run it like a tiny one, and the two favorites become an exhausted bottleneck.\nFigure 3. The spiral that leaves most of the hospital reading magazines.\nLoad balancing: gently insisting the work be spread Crucially, the model\u0026rsquo;s ordinary \u0026ldquo;get the answer right\u0026rdquo; goal does nothing to fix this, because a collapsed model can still produce reasonable answers using its few good experts. Balance has to be encouraged separately and deliberately. The fix is to add a second, mild goal alongside \u0026ldquo;be correct\u0026rdquo;: use all the experts roughly evenly. In plain terms, the model is given a nagging penalty whenever the router is overloading some experts and starving others, and that penalty pushes the router to ease off its favorites and give the neglected experts a turn. Over many rounds of training, the workload levels out. This balancing pressure — and the dials controlling how hard it pushes — is the single biggest way mixture-of-experts training differs from ordinary training. A dense model has nothing to balance, because it makes no choices. The newest designs achieve the same end even more cleanly, by keeping a running tally of how busy each expert is and quietly nudging the router toward idle ones — like a charge nurse rebalancing assignments on the fly, without ever overriding the doctors\u0026rsquo; actual judgment.\nTwo more practical headaches Because real hardware likes fixed, predictable batch sizes, each expert is given a fixed number of slots per batch — its capacity. But routing is unpredictable, so in any given batch some experts get swamped and others sit half-empty. If an expert is assigned more words than it has slots, the surplus words are simply dropped — they skip the expert and pass through unchanged, relying on the rest of the network to carry them along. There is a dial for how much headroom to give: too little and you turn patients away on busy days, too much and chairs sit empty. A well-balanced router drops very few words, and the occasional drop is survivable, but heavy dropping is a warning sign that balancing has gone wrong.\nThe second headache is communication. Because a mixture-of-experts model stores so many experts, they don\u0026rsquo;t fit on one machine, so they are spread across many — each machine holding a few. But the words being processed are also spread across machines, and the expert a given word needs probably lives on a different machine than the word does. So before the experts can run, every machine has to ship its words out to wherever the right experts are, and afterward ship all the results back. Every machine talks to every other machine, twice, for every such layer. Imagine the hospital split across a dozen buildings, with patients constantly shuttled to whichever building has their specialist and then shuttled home — and the shuttling, not the doctoring, becoming the slow part. This heavy, unavoidable communication is the defining cost of training these models, and it is why balance matters twice over: a layer can only finish as fast as its busiest expert, so an imbalanced router leaves expensive machines idle, waiting at the shuttle stop for the slowest one.\nA reality check on \u0026ldquo;experts,\u0026rdquo; and the bargain that makes it worth it It is tempting to imagine a tidy \u0026ldquo;French expert\u0026rdquo; and a \u0026ldquo;poetry expert\u0026rdquo; and a \u0026ldquo;math expert.\u0026rdquo; Reality is mushier. Specialization is real but weak and low-level: experts tend to react to surface patterns — punctuation, numbers, word types — more than to clean human topics, and responsibility for any given subject is usually smeared across several experts. The gains come less from crisp specialists than from the sheer extra capacity and the router\u0026rsquo;s freedom to mix and match a different small combination of experts for every word.\nAnd the gains are real, which is the whole justification for the complexity. At a fixed amount of computing work per word, adding more experts reliably makes the model better. In other words, mixture-of-experts lets you convert a storage budget into quality without spending more compute budget. A dense model has only one lever — spend more compute, get more of everything, and pay for all of it on every word. A mixture-of-experts model has two levers, and can buy capacity with comparatively cheap storage instead of expensive computation. The returns shrink as you pile on more experts, and the extra storage and shuttling are genuine costs, so there is a sweet spot — but that two-lever bargain is why the entire enterprise is worthwhile.\nUsing one: a small model\u0026rsquo;s speed, a large model\u0026rsquo;s footprint When the model is finally answering users, everything about how it generates text is exactly as it would be for a dense model, because none of that touches the experts. The one decisive difference is the asymmetry we have tracked all along, now in its most practical form: a mixture-of-experts model runs at the speed of a small model but demands the memory of a large one. You cannot simply keep the popular experts loaded and discard the rest, because routing is decided fresh for every word, so over any real stretch of text every expert eventually gets called. You must keep the whole hospital staffed and on-site, even though each patient sees only two doctors. This flips the usual bottleneck: a dense model is limited by raw computation, while a mixture-of-experts model is limited by memory — how much you have and how fast you can move it. The tricks for serving these models therefore all target memory, and we return to them in Chapter 11 .\nWhat to carry forward A mixture-of-experts model breaks the weld between how much a model knows and how much work it does per word. It replaces the single processing stage with a bank of experts and a tiny router that sends each word to just a few of them — so total capacity can be enormous while the per-word cost stays small. The price of this cleverness is a cluster of new problems all rooted in the router\u0026rsquo;s all-or-nothing choices: the lazy-router collapse that demands active load balancing, dropped words when experts overflow, and heavy machine-to-machine shuttling during training. And the defining trade in the end is memory for quality: cheap to compute, expensive to store.\nThat completes the family of language models — four shapes built from one toolkit. But not everything worth generating is language. In Part III we turn to a completely different way of creating, one that builds images not word by word but all at once, out of pure noise. Turn to Chapter 8 .\n← Chapter 6 · The Translators · Contents · Chapter 8 · Diffusion Models → The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/tutorials/inside-the-machines/chapter-07-mixture-of-experts/","summary":"\u003ch1 id=\"chapter-7--the-committee-mixture-of-experts\"\u003eChapter 7 — The Committee: Mixture-of-Experts\u003c/h1\u003e\n\u003cp\u003eEvery model we have met so far shares a hidden assumption: when it processes a word, it uses \u003cem\u003eall\u003c/em\u003e of itself to do it. Every one of its billions of knobs participates in every word. This is so natural that it is easy to miss as an assumption at all — but it has an expensive consequence, and the family of models in this chapter exists to escape it. These are \u003cem\u003emixture-of-experts\u003c/em\u003e models, the family that includes Switch, Mixtral, and DeepSeek, and their central idea is to build a model that is vast in what it knows while only switching on a small part of itself for any given word.\u003c/p\u003e","title":"The Committee: Mixture-of-Experts"},{"content":"Lesson 7 — The Training Algorithm in Practice Recap of Lesson 6. The noise predictor $\\epsilon_\\theta$ is an image-to-image U-Net — contracting encoder, attention bottleneck, expanding decoder, skip connections — with the timestep injected via a sinusoidal embedding passed through an MLP and added (as scale/shift) inside every residual block. Modern systems may use a Diffusion Transformer instead, but the loss is unchanged. This lesson assembles the loss and the network into the full training phase.\nEverything so far has been scaffolding — the forward jump, the tractable posterior, the bound, the great cancellation — and scaffolding can trick you into expecting an elaborate building. Here is the building. It fits on an index card: sample an image, pick a noise level, add noise, ask the network to name it, nudge the weights. Five lines. The rest of this lesson is about taking that humble loop seriously enough to make it converge to a competitive model — and about the one thing it conspicuously refuses to do.\nOverview Every result from Lessons 2–5 collapses into a loop that fits on an index card. The closed-form forward jump $q(x_t\\mid x_0)$, the tractable posterior $q(x_{t-1}\\mid x_t,x_0)$, the variational bound, and the algebraic cancellation of the KL terms into a noise-prediction MSE all exist to make this loop trivial: at no point does training simulate the reverse chain. Each gradient step performs a single one-shot noising and a single network evaluation per example. The cost asymmetry that defines diffusion — cheap training, expensive sampling — originates here.\nThe training algorithm (DDPM) $$ \\textbf{Algorithm 1: Training} $$ repeat\n$x_0 \\sim q(x_0)$ — sample a minibatch of real data $t \\sim \\mathcal{U}\\{1, \\dots, T\\}$ — sample a timestep per example, uniformly $\\epsilon \\sim \\mathcal{N}(0, I)$ — sample noise, same shape as $x_0$ $x_t = \\sqrt{\\bar\\alpha_t}\\,x_0 + \\sqrt{1-\\bar\\alpha_t}\\,\\epsilon$ — one-shot forward jump (Lesson 2) Take a gradient step on $\\nabla_\\theta \\big\\|\\epsilon - \\epsilon_\\theta(x_t, t)\\big\\|^2$ until converged\nFive lines, no adversary, no reverse rollout, no likelihood normalizer. The remainder of the lesson treats this loop as what it is: a doubly-stochastic Monte-Carlo estimator of the variational objective, and analyzes its bias, variance, and the engineering required to make it converge to a competitive model.\nFigure 1. The whole training phase, on an index card.\nThe loop as a Monte-Carlo estimator The loop is short enough to memorize in a minute; understanding why it is correct — why five lines faithfully estimate a sum over a thousand KL terms — takes a little more care.\nFrom the variational bound to $L_\\text{simple}$ Lesson 4 decomposed the negative ELBO into a sum of per-timestep KL terms,\n$$ \\mathcal L_\\text{vlb} = \\underbrace{D_{\\mathrm{KL}}!\\big(q(x_T\\mid x_0),|,p(x_T)\\big)}_{L_T}\n\\sum_{t=2}^{T}\\underbrace{D_{\\mathrm{KL}}!\\big(q(x_{t-1}\\mid x_t,x_0),|,p_\\theta(x_{t-1}\\mid x_t)\\big)}{L{t-1}} ;\\underbrace{-,\\log p_\\theta(x_0\\mid x_1)}_{L_0}. $$ With Gaussian $p_\\theta$ of fixed variance and the $\\epsilon$-parameterization, each $L_{t-1}$ reduces to a weighted noise-prediction MSE,\n$$ L_{t-1} = \\mathbb{E}_{x_0,\\epsilon}\\Big[\\, w_t \\,\\big\\|\\epsilon - \\epsilon_\\theta(x_t,t)\\big\\|^2 \\Big], \\qquad w_t = \\frac{\\beta_t^2}{2\\sigma_t^2\\,\\alpha_t\\,(1-\\bar\\alpha_t)}. $$Ho et al. (2020) observed empirically that dropping the weight ($w_t\\equiv 1$) improves sample quality, yielding the simple objective in line 5:\n$$ \\boxed{\\,L_\\text{simple}(\\theta) = \\mathbb{E}_{t\\sim\\mathcal U\\{1,T\\},\\ x_0\\sim q,\\ \\epsilon\\sim\\mathcal N(0,I)}\\Big[\\big\\|\\epsilon-\\epsilon_\\theta(x_t,t)\\big\\|^2\\Big]\\,} $$Unbiasedness and variance of the estimator The loop never forms $\\sum_t L_{t-1}$ explicitly. Instead, at each step it draws a single triple $(x_0,t,\\epsilon)$ and evaluates $g=\\big\\|\\epsilon-\\epsilon_\\theta(x_t,t)\\big\\|^2$. Because $t$ is uniform on $\\{1,\\dots,T\\}$,\n$$ \\mathbb{E}_t\\!\\left[ T\\cdot \\mathbf 1\\!\\left[\\text{term }t\\right]\\, \\big\\|\\epsilon-\\epsilon_\\theta\\big\\|^2 \\right] = \\sum_{t=1}^{T} \\mathbb{E}_{x_0,\\epsilon}\\big\\|\\epsilon-\\epsilon_\\theta(x_t,t)\\big\\|^2, $$so the per-sample gradient $\\nabla_\\theta g$ is an unbiased estimator of $\\nabla_\\theta L_\\text{simple}$ up to the constant $1/T$ absorbed into the learning rate. Minibatching of size $B$ averages $B$ such i.i.d. draws, reducing the estimator variance by $1/B$ in the usual way. The two sources of stochasticity — the data/noise draw $(x_0,\\epsilon)$ and the timestep draw $t$ — both inflate gradient variance, and the timestep variance is the one we can cheaply reshape.\nRemark (importance sampling over $t$). Replace uniform sampling by a proposal $t\\sim p(t)$ and reweight: with $p(t)\\propto \\mathbb{E}[L_{t-1}]$ optimal, the estimator $$\\hat L = \\frac{1}{p(t)}\\,L_{t-1}, \\qquad \\mathbb{E}_{t\\sim p}[\\hat L] = \\sum_t L_{t-1},$$ remains unbiased for any $p(t)\\gt 0$ but attains lower variance when $p(t)$ tracks the per-$t$ loss magnitude. Nichol \u0026amp; Dhariwal (2021) maintain a running estimate of $\\mathbb{E}[L_{t-1}^2]$ and set $p(t)\\propto\\sqrt{\\mathbb{E}[L_{t-1}^2]}$, the variance-minimizing choice for an unweighted sum. This is essential when training the full $\\mathcal L_\\text{vlb}$ (e.g. when learning $\\Sigma_\\theta$), whose per-$t$ terms span several orders of magnitude.\nRemark (Min-SNR weighting). Hang et al. (2023) reframe the choice of $w_t$ as a multi-task balancing problem across noise levels and show that $L_\\text{simple}$ ($w_t\\equiv1$) over-weights low-noise steps while $\\mathcal L_\\text{vlb}$ over-weights high-noise steps. Their Min-SNR-$\\gamma$ weight, $$ w_t = \\frac{\\min\\{\\mathrm{SNR}(t),\\,\\gamma\\}}{\\mathrm{SNR}(t)}, \\qquad \\mathrm{SNR}(t) = \\frac{\\bar\\alpha_t}{1-\\bar\\alpha_t}, $$ caps the effective gradient magnitude at high SNR (small $t$) with $\\gamma\\approx5$, accelerating convergence by 3–4$\\times$. For the $\\epsilon$-prediction target $w_t$ multiplies the loss directly; for $x_0$- or $v$-prediction the SNR factor enters differently (see the $v$-prediction remark below). Karras et al. (2022) reach the same conclusion from the continuous-time EDM perspective, prescribing a loss weighting $\\lambda(\\sigma)$ and a log-normal sampling density over noise scales $\\sigma$.\nFigure 2. Reweighting the timestep draw, without biasing it.\nLine-by-line, with the details that matter Step 1 — data. Standard minibatching. Images are scaled to $[-1,1]$ so that the empirical data standard deviation is $O(1)$ and comparable to the unit-variance noise; the variance-preserving schedule presumes signal and noise share a scale.\nRemark (data scaling and SNR). The forward marginal has variance $\\bar\\alpha_t\\,\\mathrm{Var}(x_0) + (1-\\bar\\alpha_t)$. With $\\mathrm{Var}(x_0)\\approx1$ this is $\\approx1$ for all $t$ (variance-preserving), so $\\mathrm{SNR}(t)=\\bar\\alpha_t/(1-\\bar\\alpha_t)$ is the only quantity that varies along the chain. If data are not centered and scaled — e.g. left in $[0,1]$ — the terminal $x_T$ no longer matches the standard-normal prior $p(x_T)=\\mathcal N(0,I)$, $L_T$ becomes non-negligible, and there is a train/sample prior mismatch. Latent-diffusion pipelines apply an analogous per-channel normalization to the VAE latents (a fixed scaling factor) for the same reason.\nStep 2 — sampling $t$. Each example draws an independent $t$, realizing the Monte-Carlo estimate of $\\sum_t L_{t-1}$ analyzed above. Uniform sampling corresponds to $L_\\text{simple}$; importance sampling or Min-SNR replaces it when variance reduction or rebalancing is needed.\nStep 3–4 — corruption in one shot. The single sampled $\\epsilon$ is simultaneously (a) the corruption added to $x_0$ and (b) the regression target. They are the same vector — the conceptual core of the method. Using the closed form (Lesson 2) we jump directly to noise level $t$ without simulating intermediate steps.\nStep 5 — the gradient. Plain MSE between predicted and true noise, backpropagated through $\\epsilon_\\theta$. The target is unit-variance and the loss is quadratic, so there is no min-max instability and no mode collapse.\nRemark (why we never roll out the reverse chain). The objective minimized is the variational bound, a surrogate that upper-bounds $-\\log p_\\theta(x_0)$ and decomposes into per-step KL divergences between the tractable posterior $q(x_{t-1}\\mid x_t,x_0)$ and the model $p_\\theta(x_{t-1}\\mid x_t)$. Each term is computed in closed form from a single $x_t$; matching the per-step conditionals is sufficient to control the marginal likelihood. Rolling out the reverse chain (as score-distillation or consistency-style training do) would require backpropagating through $T$ sequential network calls — quadratically more expensive and high-variance — and is provably unnecessary for the ELBO. This is the single most important structural fact of diffusion training.\nA complete training script Theory satisfied, here is what the loop looks like once every practical detail that separates a toy from a trained model has been filled back in.\nThe fragment below is a self-contained, correct training step including schedule construction (linear and cosine), buffer registration, EMA, AdamW with warmup and gradient clipping, and mixed precision. Tensor shapes are annotated; broadcasting follows PyTorch convention.\nimport math, copy import torch import torch.nn.functional as F from torch.optim import AdamW from torch.optim.lr_scheduler import LambdaLR # ---------- Schedule construction ---------- def linear_beta_schedule(T, beta_start=1e-4, beta_end=2e-2): return torch.linspace(beta_start, beta_end, T) # (T,) def cosine_alpha_bar_schedule(T, s=0.008): # Nichol \u0026amp; Dhariwal (2021): define alpha_bar directly, then derive betas. t = torch.linspace(0, T, T + 1) # (T+1,) f = torch.cos(((t / T + s) / (1 + s)) * math.pi / 2) ** 2 alpha_bar = f / f[0] # alpha_bar[0] = 1 betas = 1 - (alpha_bar[1:] / alpha_bar[:-1]) # (T,) return betas.clamp(max=0.999) # numerical guard class Diffusion(torch.nn.Module): \u0026#34;\u0026#34;\u0026#34;Holds the schedule as non-trainable buffers (move with .to(device), save with state_dict, never receive gradients).\u0026#34;\u0026#34;\u0026#34; def __init__(self, T=1000, schedule=\u0026#34;cosine\u0026#34;): super().__init__() betas = (cosine_alpha_bar_schedule(T) if schedule == \u0026#34;cosine\u0026#34; else linear_beta_schedule(T)) alphas = 1.0 - betas alpha_bars = torch.cumprod(alphas, dim=0) # (T,) self.T = T self.register_buffer(\u0026#34;betas\u0026#34;, betas) self.register_buffer(\u0026#34;alphas\u0026#34;, alphas) self.register_buffer(\u0026#34;alpha_bars\u0026#34;, alpha_bars) self.register_buffer(\u0026#34;sqrt_ab\u0026#34;, alpha_bars.sqrt()) self.register_buffer(\u0026#34;sqrt_1m_ab\u0026#34;, (1.0 - alpha_bars).sqrt()) def q_sample(self, x0, t, eps): # One-shot forward jump. x0,eps: (B,C,H,W); t: (B,) long. ab_s = self.sqrt_ab[t].view(-1, 1, 1, 1) # (B,1,1,1) ab_1m = self.sqrt_1m_ab[t].view(-1, 1, 1, 1) return ab_s * x0 + ab_1m * eps # (B,C,H,W) # ---------- EMA ---------- class EMA: \u0026#34;\u0026#34;\u0026#34;Maintains a shadow copy of parameters; use for sampling, not training.\u0026#34;\u0026#34;\u0026#34; def __init__(self, model, decay=0.9999): self.decay = decay self.shadow = copy.deepcopy(model).eval() for p in self.shadow.parameters(): p.requires_grad_(False) @torch.no_grad() def update(self, model): d = self.decay for s, p in zip(self.shadow.parameters(), model.parameters()): s.mul_(d).add_(p.detach(), alpha=1 - d) # Buffers (e.g. BN/GroupNorm stats) copied, not averaged. for sb, pb in zip(self.shadow.buffers(), model.buffers()): sb.copy_(pb) # ---------- Optimizer, warmup, grad clip, AMP ---------- def make_scheduler(optimizer, warmup_steps, total_steps): def lr_lambda(step): if step \u0026lt; warmup_steps: return step / max(1, warmup_steps) # linear warmup # cosine decay to 0 over the remaining horizon prog = (step - warmup_steps) / max(1, total_steps - warmup_steps) return 0.5 * (1 + math.cos(math.pi * prog)) return LambdaLR(optimizer, lr_lambda) def train(model, diffusion, dataloader, device, total_steps=400_000, lr=2e-4, warmup_steps=5_000, grad_clip=1.0): model.to(device); diffusion.to(device) opt = AdamW(model.parameters(), lr=lr, betas=(0.9, 0.999), weight_decay=0.0) # WD often 0 for diffusion sched = make_scheduler(opt, warmup_steps, total_steps) scaler = torch.cuda.amp.GradScaler() # mixed precision ema = EMA(model, decay=0.9999) step = 0 while step \u0026lt; total_steps: for x0 in dataloader: # x0 in [-1,1], (B,C,H,W) x0 = x0.to(device) B = x0.size(0) t = torch.randint(0, diffusion.T, (B,), device=device) # (B,) eps = torch.randn_like(x0) # label AND corruption x_t = diffusion.q_sample(x0, t, eps) # (B,C,H,W) opt.zero_grad(set_to_none=True) with torch.cuda.amp.autocast(): # fp16/bf16 forward eps_pred = model(x_t, t) # (B,C,H,W) loss = F.mse_loss(eps_pred, eps) # L_simple, one term scaler.scale(loss).backward() scaler.unscale_(opt) # unscale before clip torch.nn.utils.clip_grad_norm_(model.parameters(), grad_clip) scaler.step(opt) scaler.update() sched.step() ema.update(model) # EMA after the opt step step += 1 if step \u0026gt;= total_steps: break return ema.shadow # sample with EMA weights Numerical and shape notes.\nThe schedule buffers are 1-D (T,); indexing with t of shape (B,) gives (B,), reshaped to (B,1,1,1) for broadcasting against (B,C,H,W). cumprod in float32 is stable for $T=1000$ since each $\\alpha_t\\in(0,1)$ and $\\bar\\alpha_t$ decreases monotonically; avoid float16 for the schedule itself to prevent underflow of $\\bar\\alpha_T$. Under AMP, keep the loss reduction and the schedule in float32. GradScaler requires unscale_ before clip_grad_norm_ so the clip threshold applies to true (unscaled) gradients. EMA must run after optimizer.step(). Decay $0.999$–$0.9999$; larger decay needs more steps to \u0026ldquo;warm up\u0026rdquo; the shadow but yields smoother samples. Set weight_decay=0 (or very small) — diffusion U-Nets are typically not regularized this way; the noise target already regularizes. $v$-prediction and alternative targets The network may regress a different target without changing the loop:\n$$ \\hat x_0\\text{-prediction:}\\quad \\|x_0 - x_{0,\\theta}\\|^2, \\qquad v\\text{-prediction:}\\quad \\|v - v_\\theta\\|^2,\\quad v \\equiv \\sqrt{\\bar\\alpha_t}\\,\\epsilon - \\sqrt{1-\\bar\\alpha_t}\\,x_0. $$ Remark (when to use $v$). The $\\epsilon$-target degenerates as $t\\to0$ (where almost no noise is present, so $\\epsilon$ is barely identifiable) and the $x_0$-target degenerates as $t\\to T$ (pure noise, $x_0$ unidentifiable). The $v$-parameterization of Salimans \u0026amp; Ho (2022) interpolates: $v$ is a unit-variance, well-conditioned target across all noise levels, equal to $\\epsilon$ at $t=T$ and to $-x_0$-direction at $t=0$. This makes $v$-prediction the preferred target for (i) progressive distillation, where the same network is re-trained at fewer steps and must behave well near $t=0$, (ii) high-resolution and terminal-SNR-zero schedules where $\\bar\\alpha_T\\to0$ exactly, and (iii) any regime where Min-SNR-style weighting is applied. Conversion between targets is exact and algebraic, so a trained $\\epsilon_\\theta$ can be reinterpreted at sampling time without retraining.\nWhat we are not doing during training It bears repeating because it is the most counterintuitive part: the reverse chain is never executed during training. No images are generated to compute the loss; each gradient step touches exactly one noise level per example. Generation — the sequential, expensive part — happens only at inference (Lesson 8). The training objective is a surrogate (the variational bound) that guarantees a good reverse model without ever running it.\nFigure 3. The most counterintuitive fact in the method.\nExercises 1. Unbiasedness of the loop. With $t\\sim\\mathcal U\\{1,\\dots,T\\}$, show the single-draw loss $g=\\|\\epsilon-\\epsilon_\\theta(x_t,t)\\|^2$ has expectation $\\frac1T\\sum_{t=1}^T\\mathbb E_{x_0,\\epsilon}\\|\\epsilon-\\epsilon_\\theta(x_t,t)\\|^2$, so $\\nabla_\\theta g$ is an unbiased estimator of $\\nabla_\\theta L_\\text{simple}$ up to the constant $1/T$.\nLet $f(t)=\\mathbb E_{x_0,\\epsilon}\\|\\epsilon-\\epsilon_\\theta(x_t,t)\\|^2$. Since $t$ is uniform, $\\mathbb E_{t,x_0,\\epsilon}[g]=\\sum_{t=1}^T\\frac1T f(t)=\\frac1T\\sum_t f(t)$. Gradients commute with the (finite) expectation: $\\mathbb E[\\nabla_\\theta g]=\\frac1T\\sum_t\\nabla_\\theta f(t)=\\frac1T\\nabla_\\theta\\sum_t f(t)$. The factor $1/T$ is a fixed scalar absorbed into the learning rate, so the per-sample gradient is an unbiased estimator of $\\nabla_\\theta L_\\text{simple}$ (which uses $\\frac1T\\sum_t$ via the uniform $t$).\n2. Importance sampling stays unbiased. For any proposal $p(t)\\gt 0$ and the estimator $\\hat L=L_{t-1}/p(t)$, show $\\mathbb E_{t\\sim p}[\\hat L]=\\sum_t L_{t-1}$. What choice of $p(t)$ minimizes variance?\n$\\mathbb E_{t\\sim p}[\\hat L]=\\sum_t p(t)\\cdot\\frac{L_{t-1}}{p(t)}=\\sum_t L_{t-1}$ — unbiased for any $p(t)\\gt 0$. Variance is reduced when $p(t)$ tracks the per-$t$ loss magnitude; for an unweighted sum the variance-minimizing proposal is $p(t)\\propto\\sqrt{\\mathbb E[L_{t-1}^2]}$ (Nichol \u0026amp; Dhariwal maintain a running estimate of $\\mathbb E[L_{t-1}^2]$).\n3. Why data must be scaled to $[-1,1]$. If images are left in $[0,1]$ (so $\\mathbb E[x_0]\\approx0.5$, $\\mathrm{Var}(x_0)\\ne1$), what breaks at the terminal step, and why does it make $L_T$ non-negligible?\nThe forward marginal has mean $\\sqrt{\\bar\\alpha_t}\\,\\mathbb E[x_0]$ and variance $\\bar\\alpha_t\\mathrm{Var}(x_0)+(1-\\bar\\alpha_t)$. The variance-preserving design relies on $\\mathrm{Var}(x_0)\\approx1$ to keep this $\\approx1$ for all $t$. With un-centered, un-scaled data the terminal $q(x_T)$ no longer matches the standard-normal prior $p(x_T)=\\mathcal N(0,I)$, so $L_T=D_{\\mathrm{KL}}(q(x_T\\mid x_0)\\,\\|\\,\\mathcal N(0,I))$ becomes non-negligible and there is a train/sample prior mismatch (sampling starts from $\\mathcal N(0,I)$, which the chain was not trained to land on). Centering and scaling to $[-1,1]$ restores the match.\n4. EMA ordering. Why must ema.update(model) run after optimizer.step(), and what are the EMA weights used for?\nEMA maintains a running average of the post-update parameters: $s\\leftarrow d\\,s+(1-d)\\,\\theta_{\\text{new}}$. Calling it before the optimizer step would average stale, pre-update weights and lag the shadow by a step. The EMA shadow is used at sampling time (not for the gradient step) because the averaged weights give noticeably smoother, higher-quality samples.\nConclusion Training is a five-line loop that is best understood as a doubly-stochastic Monte-Carlo estimator of $\\sum_t L_{t-1}$: sample data, sample a timestep (uniform, importance-sampled, or Min-SNR-weighted), sample noise, jump to that noise level in one shot, and regress the network\u0026rsquo;s prediction onto the true target with MSE. The estimator is unbiased; its variance is controlled by batch size and by the timestep proposal. Practical convergence to a competitive model requires EMA weights, a cosine schedule, AdamW with warmup and gradient clipping, mixed precision, and consistent $[-1,1]$ data scaling — and optionally a $v$-prediction target when the SNR endpoints matter. None of this ever rolls out the reverse process. With $\\epsilon_\\theta$ (or $v_\\theta$) trained, Lesson 8 finally runs the reverse chain: the DDPM sampling algorithm that turns pure noise into a sample.\nThe index card is written, and what it never does — run the reverse chain even once — is precisely the thing Lesson 8 does for the first time. We have a trained noise predictor; now we make it dream.\nReferences J. Ho, A. Jain, P. Abbeel. Denoising Diffusion Probabilistic Models. NeurIPS 2020. A. Nichol, P. Dhariwal. Improved Denoising Diffusion Probabilistic Models. ICML 2021. (Cosine schedule; importance sampling over $t$; learned variances.) T. Salimans, J. Ho. Progressive Distillation for Fast Sampling of Diffusion Models. ICLR 2022. ($v$-prediction.) T. Karras, M. Aittala, T. Aila, S. Laine. Elucidating the Design Space of Diffusion-Based Generative Models (EDM). NeurIPS 2022. (Loss weighting and noise-level sampling.) T. Hang et al. Efficient Diffusion Training via Min-SNR Weighting Strategy. ICCV 2023. D. Kingma, T. Salimans, B. Poole, J. Ho. Variational Diffusion Models. NeurIPS 2021. (SNR-based formulation of the bound.) ← Previous: Lesson 6 — The U-Net Architecture · Course Index · Next: Lesson 8 — DDPM Sampling → · Appendix A: Acronyms · Appendix B: Formulas The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/tutorials/diffusion-lessons/lesson-07-training-algorithm/","summary":"\u003ch1 id=\"lesson-7--the-training-algorithm-in-practice\"\u003eLesson 7 — The Training Algorithm in Practice\u003c/h1\u003e\n\u003cblockquote\u003e\n\u003cp\u003e\u003cstrong\u003eRecap of Lesson 6.\u003c/strong\u003e The noise predictor $\\epsilon_\\theta$ is an image-to-image U-Net — contracting encoder, attention bottleneck, expanding decoder, skip connections — with the timestep injected via a sinusoidal embedding passed through an MLP and added (as scale/shift) inside every residual block. Modern systems may use a Diffusion Transformer instead, but the loss is unchanged. This lesson assembles the loss and the network into the full \u003cstrong\u003etraining phase\u003c/strong\u003e.\u003c/p\u003e","title":"Training Algorithm"},{"content":"Lesson 8 — The Sampling Algorithm: Inference with DDPM Recap of Lesson 7. Training is a five-line loop: sample $x_0$, sample $t\\sim\\mathcal U\\{1,T\\}$, sample $\\epsilon\\sim\\mathcal N(0,I)$, form $x_t=\\sqrt{\\bar\\alpha_t}x_0+\\sqrt{1-\\bar\\alpha_t}\\epsilon$ in one shot, and minimize $\\|\\epsilon-\\epsilon_\\theta(x_t,t)\\|^2$. Crucially, training never runs the reverse chain; it uses the variational surrogate. EMA weights and a good schedule make it stable. Now we finally run the reverse chain to generate.\nUntil now the reverse process has been a thing we reasoned about but never ran. This lesson finally turns the crank. We begin with a square of pure static — no image, nothing, just a draw from a Gaussian — and put the same patient question to the trained network a thousand times over: what is the noise here, and what lies one notch beneath it? Slowly, out of the static, a face that never existed sharpens into view. The price of that thousand-fold patience is the very problem the next lesson exists to solve.\nOverview We have a trained noise predictor $\\epsilon_\\theta$ (use the EMA weights from Lesson 7). Generation inverts the forward corruption: initialize a canvas $x_T\\sim\\mathcal N(0,I)$ — matching the prior the forward process converges to — and iterate the learned reverse kernel down to $x_0$. Each step estimates the noise in the current state, removes a calibrated fraction of it, and re-injects a controlled amount of fresh randomness so that we draw from the conditional distribution rather than collapsing to its mode. After $T$ steps a coherent sample emerges. Unlike training, inference is inherently sequential, and that sequentiality is the central efficiency problem this lesson sets up.\nThe DDPM sampling step We sample from the learned reverse Gaussian $p_\\theta(x_{t-1}\\mid x_t)=\\mathcal N\\!\\big(\\mu_\\theta(x_t,t),\\sigma_t^2 I\\big)$. Using the $\\epsilon$-parameterized mean from Lesson 5, the update is:\n$$ \\boxed{\\ x_{t-1} = \\frac{1}{\\sqrt{\\alpha_t}}\\left(x_t - \\frac{1-\\alpha_t}{\\sqrt{1-\\bar\\alpha_t}}\\,\\epsilon_\\theta(x_t, t)\\right) + \\sigma_t\\, z, \\qquad z\\sim\\mathcal N(0,I)\\ } $$with $z = 0$ at the final step $t=1$ (we output the mean). Two standard variance choices work comparably well:\n$$ \\sigma_t^2 = \\beta_t \\qquad\\text{or}\\qquad \\sigma_t^2 = \\tilde\\beta_t = \\frac{1-\\bar\\alpha_{t-1}}{1-\\bar\\alpha_t}\\,\\beta_t. $$Derivation of the mean The tractable forward posterior (Lesson 3) is Gaussian,\n$$ q(x_{t-1}\\mid x_t,x_0) = \\mathcal N\\!\\big(\\tilde\\mu_t(x_t,x_0),\\,\\tilde\\beta_t I\\big), \\qquad \\tilde\\mu_t(x_t,x_0) = \\frac{\\sqrt{\\bar\\alpha_{t-1}}\\,\\beta_t}{1-\\bar\\alpha_t}\\,x_0 + \\frac{\\sqrt{\\alpha_t}\\,(1-\\bar\\alpha_{t-1})}{1-\\bar\\alpha_t}\\,x_t. $$The training objective drives $p_\\theta$ to match $q(x_{t-1}\\mid x_t,x_0)$, so we set $\\mu_\\theta = \\tilde\\mu_t$ but with the unknown $x_0$ replaced by the estimate the network implies. Inverting the forward jump $x_t=\\sqrt{\\bar\\alpha_t}x_0+\\sqrt{1-\\bar\\alpha_t}\\epsilon$ gives\n$$ \\hat x_0 = \\frac{1}{\\sqrt{\\bar\\alpha_t}}\\big(x_t - \\sqrt{1-\\bar\\alpha_t}\\,\\epsilon_\\theta(x_t,t)\\big). $$Substituting $\\hat x_0$ into $\\tilde\\mu_t$ and simplifying with $\\alpha_t=1-\\beta_t$ and $\\bar\\alpha_t=\\alpha_t\\bar\\alpha_{t-1}$ collapses the two terms into the single $\\epsilon$-form\n$$ \\mu_\\theta(x_t,t) = \\frac{1}{\\sqrt{\\alpha_t}}\\left(x_t - \\frac{1-\\alpha_t}{\\sqrt{1-\\bar\\alpha_t}}\\,\\epsilon_\\theta(x_t,t)\\right), $$which is the deterministic part of the boxed update. The reduction is purely algebraic; the cross-terms in $x_t$ cancel because $\\sqrt{\\alpha_t}(1-\\bar\\alpha_{t-1}) + \\beta_t/\\sqrt{\\alpha_t}\\cdot\\bar\\alpha_{t-1}\\cdots$ regroup to $1/\\sqrt{\\alpha_t}$ times the bracket.\nWhy $\\sigma_t^2\\in\\{\\beta_t,\\tilde\\beta_t\\}$ Remark (the two variances are bounds, Improved DDPM). With the mean fixed to $\\tilde\\mu_t$, the optimal reverse variance lies between two extremes. If the data were a single point ($x_0$ deterministic given $x_t$), the reverse conditional variance is exactly the posterior variance $\\tilde\\beta_t = \\tfrac{1-\\bar\\alpha_{t-1}}{1-\\bar\\alpha_t}\\beta_t$ (lower bound). If the data distribution is the standard normal, the reverse variance equals the forward variance $\\beta_t$ (upper bound). One checks $\\tilde\\beta_t \\le \\beta_t$ for all $t$, with the gap small for most $t$ but large near $t=1$. Ho et al. (2020) found both endpoints give similar sample quality. Nichol \u0026amp; Dhariwal (2021) instead learn an interpolation in log-space, $$\\sigma_t^2 = \\exp\\!\\big(v\\log\\beta_t + (1-v)\\log\\tilde\\beta_t\\big),\\qquad v=v_\\theta(x_t,t)\\in[0,1],$$ trained with the full $\\mathcal L_\\text{vlb}$ (the simple loss is independent of the variance), which improves log-likelihood and enables high-quality sampling in far fewer steps.\nReading the update Deterministic denoising (the mean). $\\mu_\\theta$ removes the network\u0026rsquo;s noise estimate and rescales — it is the posterior mean $\\tilde\\mu_t$ with $\\epsilon$ replaced by $\\epsilon_\\theta$, i.e. with $x_0$ replaced by $\\hat x_0$. Stochastic injection. $\\sigma_t z$ re-adds randomness because $p_\\theta(x_{t-1}\\mid x_t)$ is a distribution, not a point. Remark (why re-inject noise). Taking the mean at every step would push the chain toward the mode of each conditional and, compounded over $T$ steps, toward an over-smoothed average image — a single high-probability blur rather than a sample. The forward process is genuinely stochastic; faithfully inverting it requires sampling $x_{t-1}\\sim p_\\theta(\\cdot\\mid x_t)$, which the $\\sigma_t z$ term implements. The exception is the final step ($t=1$): there is no $x_0$ noise to model, so we emit the mean, $z=0$, avoiding an unnecessary variance injection on the output.\nFigure 1. One reverse step: denoise, then re-randomize.\nThe full sampling algorithm (DDPM) With the single step understood, the full procedure is simply that step, repeated with discipline from pure noise all the way down to a picture.\n$$ \\textbf{Algorithm 2: Sampling} $$ $x_T \\sim \\mathcal{N}(0, I)$ — start from pure noise for $t = T, T-1, \\dots, 1$ do $z \\sim \\mathcal{N}(0, I)$ if $t \\gt 1$, else $z = 0$ $x_{t-1} = \\dfrac{1}{\\sqrt{\\alpha_t}}\\left(x_t - \\dfrac{1-\\alpha_t}{\\sqrt{1-\\bar\\alpha_t}}\\,\\epsilon_\\theta(x_t, t)\\right) + \\sigma_t z$ end for return $x_0$ Index convention (read once, then the code reads cleanly). The math indexes timesteps $t=1,\\dots,T$; PyTorch arrays are 0-based, so the schedule buffers store level $t$ at position $t-1$. The loop variable i is the array position, and it runs downward T-1, …, 0. The single rule that resolves every cross-reference is:\n$$ \\texttt{i} \\;=\\; t-1 \\qquad\\Longleftrightarrow\\qquad t \\;=\\; \\texttt{i}+1. $$ loop i math $t$ alpha_bars[i] alpha_bars[i-1] noise? T-1 $T$ $\\bar\\alpha_T$ $\\bar\\alpha_{T-1}$ yes … … … … yes 1 $2$ $\\bar\\alpha_2$ $\\bar\\alpha_1$ yes 0 $1$ $\\bar\\alpha_1$ $\\bar\\alpha_0:=1$ (guarded) no ($z=0$) So alpha_bars[i] is $\\bar\\alpha_t$ and alpha_bars[i-1] is $\\bar\\alpha_{t-1}$; the terminal iteration i=0 is the $t=1$ step that emits the mean. Whenever the code says [i], mentally read $\\bar\\alpha_t$; whenever it says [i-1], read $\\bar\\alpha_{t-1}$.\nA complete, correct implementation supporting both $\\sigma_t$ choices, optional $\\hat x_0$ clamping / dynamic thresholding, and the $z=0$ terminal step:\n@torch.no_grad() def ddpm_sample(model, shape, diffusion, device, sigma_kind=\u0026#34;tilde_beta\u0026#34;, # \u0026#34;beta\u0026#34; or \u0026#34;tilde_beta\u0026#34; thresholding=\u0026#34;clamp\u0026#34;, # None | \u0026#34;clamp\u0026#34; | \u0026#34;dynamic\u0026#34; dynamic_p=0.995): \u0026#34;\u0026#34;\u0026#34; model: trained EMA network, returns eps_theta(x_t, t), shape (B,C,H,W) diffusion: holds buffers betas, alphas, alpha_bars, all shape (T,) \u0026#34;\u0026#34;\u0026#34; betas, alphas, alpha_bars = diffusion.betas, diffusion.alphas, diffusion.alpha_bars T = diffusion.T x = torch.randn(shape, device=device) # x_T ~ N(0, I) for i in reversed(range(T)): # i = T-1, ..., 0 (=\u0026gt; t = i+1) t_batch = torch.full((shape[0],), i, device=device, dtype=torch.long) eps = model(x, t_batch) # (B,C,H,W) a = alphas[i] # scalar tensors ab = alpha_bars[i] b = betas[i] ab_prev = alpha_bars[i - 1] if i \u0026gt; 0 else torch.ones((), device=device) # Predict x0, optionally constrain it (improves high-guidance stability). x0_hat = (x - (1 - ab).sqrt() * eps) / ab.sqrt() # (B,C,H,W) if thresholding == \u0026#34;clamp\u0026#34;: x0_hat = x0_hat.clamp(-1.0, 1.0) elif thresholding == \u0026#34;dynamic\u0026#34;: # Imagen-style s = torch.quantile(x0_hat.abs().flatten(1), dynamic_p, dim=1) s = s.clamp(min=1.0).view(-1, 1, 1, 1) # threshold \u0026gt;= 1 x0_hat = x0_hat.clamp(-s, s) / s # Posterior mean tilde_mu_t(x_t, x0_hat); equivalent to the boxed eps-form. mean = (ab_prev.sqrt() * b / (1 - ab)) * x0_hat \\ + (a.sqrt() * (1 - ab_prev) / (1 - ab)) * x if i \u0026gt; 0: if sigma_kind == \u0026#34;beta\u0026#34;: var = b else: # tilde_beta (lower bound) var = (1 - ab_prev) / (1 - ab) * b x = mean + var.sqrt() * torch.randn_like(x) # x_{t-1} else: x = mean # t = 1: no noise injected return x # x_0, a fresh sample Shape and correctness notes.\nLoop index i runs T-1 … 0, corresponding to timestep t = i+1; the terminal iteration i=0 is the $t=1$ step where $z=0$. alpha_bars[i-1] is undefined at i=0; we use $\\bar\\alpha_{-1}\\equiv1$ there, but since that branch emits the mean only, the value is irrelevant — it is guarded so the deterministic-view code below is exact. Schedule buffers stay float32 to keep $\\sqrt{1-\\bar\\alpha_t}$ stable for large $T$; cast x/eps as needed for AMP but reduce in float32. @torch.no_grad() is mandatory: no gradients are needed and it halves memory, allowing larger batches. Figure 2. The one index rule that resolves every cross-reference.\nAn equivalent \u0026ldquo;predict $\\hat x_0$, then re-noise\u0026rdquo; view The code above already uses this view; here is why it equals the boxed update. At step $t$:\nEstimate the clean image by inverting the forward jump: $$ \\hat x_0 = \\frac{1}{\\sqrt{\\bar\\alpha_t}}\\big(x_t - \\sqrt{1-\\bar\\alpha_t}\\,\\epsilon_\\theta(x_t,t)\\big). $$ Re-noise to level $t-1$ by sampling $q(x_{t-1}\\mid x_t,\\hat x_0)=\\mathcal N\\!\\big(\\tilde\\mu_t(x_t,\\hat x_0),\\tilde\\beta_t I\\big)$. Proof of equivalence. Insert $\\hat x_0$ into the posterior mean:\n$$ \\tilde\\mu_t(x_t,\\hat x_0) = \\frac{\\sqrt{\\bar\\alpha_{t-1}},\\beta_t}{1-\\bar\\alpha_t},\\hat x_0\n\\frac{\\sqrt{\\alpha_t},(1-\\bar\\alpha_{t-1})}{1-\\bar\\alpha_t},x_t. $$ Substitute $\\hat x_0 = \\big(x_t-\\sqrt{1-\\bar\\alpha_t}\\,\\epsilon_\\theta\\big)/\\sqrt{\\bar\\alpha_t}$ and use $\\bar\\alpha_t=\\alpha_t\\bar\\alpha_{t-1}$ so that $\\sqrt{\\bar\\alpha_{t-1}}/\\sqrt{\\bar\\alpha_t} = 1/\\sqrt{\\alpha_t}$. The coefficient of $x_t$ becomes\n$$ \\frac{\\beta_t}{(1-\\bar\\alpha_t)\\sqrt{\\alpha_t}} + \\frac{\\sqrt{\\alpha_t}(1-\\bar\\alpha_{t-1})}{1-\\bar\\alpha_t} = \\frac{\\beta_t + \\alpha_t(1-\\bar\\alpha_{t-1})}{(1-\\bar\\alpha_t)\\sqrt{\\alpha_t}} = \\frac{1-\\bar\\alpha_t}{(1-\\bar\\alpha_t)\\sqrt{\\alpha_t}} = \\frac{1}{\\sqrt{\\alpha_t}}, $$using $\\beta_t+\\alpha_t(1-\\bar\\alpha_{t-1}) = 1-\\alpha_t+\\alpha_t-\\alpha_t\\bar\\alpha_{t-1} = 1-\\bar\\alpha_t$. The coefficient of $\\epsilon_\\theta$ is\n$$ -\\frac{\\sqrt{\\bar\\alpha_{t-1}}\\,\\beta_t}{(1-\\bar\\alpha_t)\\sqrt{\\bar\\alpha_t}}\\sqrt{1-\\bar\\alpha_t} = -\\frac{\\beta_t}{\\sqrt{\\alpha_t}\\,\\sqrt{1-\\bar\\alpha_t}} = -\\frac{1}{\\sqrt{\\alpha_t}}\\cdot\\frac{1-\\alpha_t}{\\sqrt{1-\\bar\\alpha_t}}. $$Collecting the two coefficients yields exactly $\\mu_\\theta=\\frac{1}{\\sqrt{\\alpha_t}}\\big(x_t-\\frac{1-\\alpha_t}{\\sqrt{1-\\bar\\alpha_t}}\\epsilon_\\theta\\big)$, the deterministic part of the boxed update; with variance $\\tilde\\beta_t$ this is line 4 verbatim. $\\;\\blacksquare$\nConceptually: each step makes a full guess of the final image, then steps one notch back toward it. Early $\\hat x_0$ is a blurry hallucination and sharpens as $t$ decreases.\nRemark (clamping and dynamic thresholding). Because $\\hat x_0$ is exposed explicitly, we can constrain it to the valid data range $[-1,1]$. Naive clamping helps at high classifier-free-guidance scales where the unconstrained $\\hat x_0$ saturates. Saharia et al. (2022, Imagen) replace static clamping with dynamic thresholding: choose the per-image threshold $s$ as the $p$-th percentile of $|\\hat x_0|$ (with $s\\ge1$), clip to $[-s,s]$, then rescale by $1/s$. This prevents the oversaturation and color shifts that strong guidance otherwise causes, and is now standard in high-resolution text-to-image samplers (Lesson 10).\nFigure 3. The same step, and what the second reading unlocks.\nThe cost of inference, and why it motivates Lesson 9 The sampler works — and in working, it hands us the bill that the rest of the course is devoted to paying down.\nRemark (sequential cost and discretization error). Sampling calls the network once per step, and DDPM uses $T\\approx1000$ steps, so one image needs ~1000 sequential forward passes — far slower than a GAN\u0026rsquo;s single pass. This is the practical weakness of diffusion. The deeper framing: the boxed update is the Euler–Maruyama discretization of a continuous-time reverse SDE, and $T$ is just the number of integration steps. The per-step error is a discretization error that vanishes as the step size shrinks; conversely, with a better integrator we can take far larger steps. Since $\\epsilon_\\theta$ was trained on a continuum of noise levels via the closed-form jump, inference is free to use a coarse, non-uniform subsequence of timesteps and even a deterministic ($\\sigma_t=0$) update. That is exactly what DDIM exploits, and it opens the score-based / SDE view that unifies diffusion with stochastic calculus.\nExercises 1. Coefficient collapse. Substitute $\\hat x_0=\\frac{1}{\\sqrt{\\bar\\alpha_t}}(x_t-\\sqrt{1-\\bar\\alpha_t}\\,\\epsilon_\\theta)$ into the posterior mean $\\tilde\\mu_t(x_t,\\hat x_0)$ and show the coefficient of $x_t$ equals $1/\\sqrt{\\alpha_t}$.\nUsing $\\sqrt{\\bar\\alpha_{t-1}}/\\sqrt{\\bar\\alpha_t}=1/\\sqrt{\\alpha_t}$, the $x_t$-coefficient is\n$$ \\frac{\\beta_t}{(1-\\bar\\alpha_t)\\sqrt{\\alpha_t}}+\\frac{\\sqrt{\\alpha_t}(1-\\bar\\alpha_{t-1})}{1-\\bar\\alpha_t} =\\frac{\\beta_t+\\alpha_t(1-\\bar\\alpha_{t-1})}{(1-\\bar\\alpha_t)\\sqrt{\\alpha_t}} =\\frac{1-\\bar\\alpha_t}{(1-\\bar\\alpha_t)\\sqrt{\\alpha_t}}=\\frac{1}{\\sqrt{\\alpha_t}}, $$since $\\beta_t+\\alpha_t(1-\\bar\\alpha_{t-1})=1-\\alpha_t+\\alpha_t-\\alpha_t\\bar\\alpha_{t-1}=1-\\bar\\alpha_t$.\n2. The terminal step. Why do we set $z=0$ (emit the mean) at $t=1$ rather than injecting $\\sigma_1 z$?\nAt the last step we sample $p_\\theta(x_0\\mid x_1)$ to produce the clean output; there is no further $x_0$-level noise to model, and the true posterior variance $\\tilde\\beta_1=0$. Emitting the mean avoids adding an unnecessary, quality-degrading variance injection to the final image.\n3. Index sanity check. Using the index convention, at loop iteration i=0: what is the math timestep $t$, what value is used for $\\bar\\alpha_{t-1}$, and why is that value irrelevant to the output?\ni=0 $\\Rightarrow t=i+1=1$. Then $\\bar\\alpha_{t-1}=\\bar\\alpha_0:=1$ (the guarded value torch.ones(...)). It is irrelevant because at $t=1$ the code takes the no-noise branch and emits the mean ($z=0$); $\\bar\\alpha_{t-1}$ only ever enters the variance term, which is not used at this step.\n4. The two variances. State the two standard choices of $\\sigma_t^2$, say which is the lower and which the upper bound on the optimal reverse variance, and where the gap between them is largest.\n$\\sigma_t^2=\\tilde\\beta_t=\\frac{1-\\bar\\alpha_{t-1}}{1-\\bar\\alpha_t}\\beta_t$ is the lower bound (exact when the data is a single point given $x_t$); $\\sigma_t^2=\\beta_t$ is the upper bound (exact when the data is standard normal). One checks $\\tilde\\beta_t\\le\\beta_t$, with the gap small for most $t$ but largest near $t=1$. Ho et al. found both give comparable sample quality; Improved DDPM learns an interpolation between them.\nConclusion Inference runs the learned reverse chain: start at $x_T\\sim\\mathcal N(0,I)$ and iterate the DDPM update, which subtracts the network\u0026rsquo;s predicted noise (the deterministic posterior mean $\\mu_\\theta=\\tilde\\mu_t$ with $x_0\\to\\hat x_0$) and re-injects $\\sigma_t z$ — with $\\sigma_t^2$ either $\\beta_t$ (upper bound) or $\\tilde\\beta_t$ (lower bound), the two extremes that Improved DDPM interpolates by learning. The \u0026ldquo;predict $\\hat x_0$, re-noise to $t-1$\u0026rdquo; view is algebraically identical to the boxed update and makes thresholding natural. The stochastic term is what samples from the distribution rather than its mode; the price is ~$T$ sequential network calls, a discretization of an underlying reverse SDE. In Lesson 9 we attack that cost with DDIM (fewer, deterministic steps) and reveal the deep score-based / SDE view.\nWe turned the crank and, a thousand steps later, a sample appeared. The obvious next question is whether we truly needed all thousand — and Lesson 9, in answering \u0026ldquo;no,\u0026rdquo; uncovers what the chain was all along: the clumsy discretization of a smooth, continuous flow.\nReferences J. Ho, A. Jain, P. Abbeel. Denoising Diffusion Probabilistic Models. NeurIPS 2020. (Reverse update; $\\sigma_t^2\\in\\{\\beta_t,\\tilde\\beta_t\\}$.) A. Nichol, P. Dhariwal. Improved Denoising Diffusion Probabilistic Models. ICML 2021. (Learned interpolated variance; few-step sampling.) J. Song, C. Meng, S. Ermon. Denoising Diffusion Implicit Models (DDIM). ICLR 2021. (Deterministic, accelerated sampling — Lesson 9.) C. Saharia et al. Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding (Imagen). NeurIPS 2022. (Dynamic thresholding.) ← Previous: Lesson 7 — The Training Algorithm · Course Index · Next: Lesson 9 — DDIM \u0026amp; the Score View → · Appendix A: Acronyms · Appendix B: Formulas The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/tutorials/diffusion-lessons/lesson-08-ddpm-sampling/","summary":"\u003ch1 id=\"lesson-8--the-sampling-algorithm-inference-with-ddpm\"\u003eLesson 8 — The Sampling Algorithm: Inference with DDPM\u003c/h1\u003e\n\u003cblockquote\u003e\n\u003cp\u003e\u003cstrong\u003eRecap of Lesson 7.\u003c/strong\u003e Training is a five-line loop: sample $x_0$, sample $t\\sim\\mathcal U\\{1,T\\}$, sample $\\epsilon\\sim\\mathcal N(0,I)$, form $x_t=\\sqrt{\\bar\\alpha_t}x_0+\\sqrt{1-\\bar\\alpha_t}\\epsilon$ in one shot, and minimize $\\|\\epsilon-\\epsilon_\\theta(x_t,t)\\|^2$. Crucially, training never runs the reverse chain; it uses the variational surrogate. EMA weights and a good schedule make it stable. Now we finally \u003cstrong\u003erun the reverse chain\u003c/strong\u003e to generate.\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003chr\u003e\n\u003cblockquote\u003e\n\u003cp\u003e\u003cem\u003eUntil now the reverse process has been a thing we reasoned about but never ran. This lesson finally turns the crank. We begin with a square of pure static — no image, nothing, just a draw from a Gaussian — and put the same patient question to the trained network a thousand times over: what is the noise here, and what lies one notch beneath it? Slowly, out of the static, a face that never existed sharpens into view. The price of that thousand-fold patience is the very problem the next lesson exists to solve.\u003c/em\u003e\u003c/p\u003e","title":"DDPM Sampling"},{"content":"Lesson 8 — Optimizers, Learning Rate \u0026amp; Batches Technical course on LLM mechanics — Lesson 8 of 10.\nRecap of Lesson 7 Backprop computes $\\nabla_\\theta\\mathcal L\\in\\mathbb R^P$ in one reverse sweep via VJPs ($\\bar W=\\bar{\\mathbf u}\\mathbf x^\\top$, $\\bar{\\mathbf x}=W^\\top\\bar{\\mathbf u}$), seeded by $\\hat{\\mathbf p}-\\mathbf y$. The bare update is $\\theta\\leftarrow\\theta-\\eta\\nabla_\\theta\\mathcal L$. We estimate the gradient on mini-batches (unbiased, variance $\\propto1/|B|$). Residuals/normalization/scaling keep the Jacobian product conditioned; checkpointing tames memory. This lesson specifies the optimizer that maps $g\\mapsto$ update, the schedule on $\\eta$, and the regularizers — all [CONFIG], plus the optimizer\u0026rsquo;s [STATE].\nLesson 7 handed us a direction; it deliberately stopped short of saying how far to step. That gap is the whole subject here. The boxed update $\\theta\\leftarrow\\theta-\\eta\\nabla_\\theta\\mathcal L$ is correct in principle and almost unusable in practice at scale — so the second half of the engine room replaces it with the machinery that actually trains a frontier model: an optimizer that reshapes the step coordinate by coordinate, a schedule that moves $\\eta$ over the run, and the planning laws that size the whole thing before a single step is taken.\n1. Why plain SGD is insufficient at scale $\\theta\\leftarrow\\theta-\\eta g$ has two weaknesses on LLM loss landscapes: (i) a single scalar $\\eta$ for all $P$ coordinates is wrong when gradient magnitudes vary by orders of magnitude across parameters; (ii) the curvature is ill-conditioned (the Hessian\u0026rsquo;s condition number is huge), so a step size safe in steep directions crawls in flat ones, and SGD oscillates across ravines. The fixes are momentum (use gradient history) and per-coordinate adaptive scaling (Adam). We build them up in that order, because Adam is essentially momentum plus one more idea.\nFigure 1. From SGD to AdamW, one idea at a time.\n2. Momentum Maintain an exponential moving average (EMA) of gradients and step on it:\n$$ \\mathbf m_t=\\beta\\,\\mathbf m_{t-1}+(1-\\beta)g_t,\\qquad \\theta\\leftarrow\\theta-\\eta\\,\\mathbf m_t. $$With $\\beta\\approx0.9$, $\\mathbf m_t$ averages roughly the last $1/(1-\\beta)=10$ gradients. Consistent directions accumulate; oscillating components cancel. This is a low-pass filter on the gradient that both accelerates along ravines and damps noise. $\\beta$ is [CONFIG]; $\\mathbf m$ is [STATE] ($P$ extra numbers).\nMomentum fixes the direction but still applies a single $\\eta$ to every coordinate — and that was the other half of the problem with plain SGD. Adam keeps the gradient-averaging idea and adds a per-coordinate rescaling on top of it.\n3. Adam / AdamW — the LLM workhorse Adam keeps two EMAs per parameter — first moment (mean, $\\mathbf m$) and second raw moment (uncentered variance, $\\mathbf v$):\n$$ \\mathbf m_t=\\beta_1\\mathbf m_{t-1}+(1-\\beta_1)g_t,\\qquad \\mathbf v_t=\\beta_2\\mathbf v_{t-1}+(1-\\beta_2)g_t^{\\odot2}. $$Bias correction — the derivation Initialized at $\\mathbf m_0=\\mathbf v_0=0$, the EMAs are biased toward zero early on. Unrolling and assuming $g$ roughly stationary with mean $\\mathbb E[g]$:\n$$ \\mathbf m_t=(1-\\beta_1)\\sum_{i=1}^{t}\\beta_1^{\\,t-i}g_i\\ \\Rightarrow\\ \\mathbb E[\\mathbf m_t]=\\mathbb E[g]\\,(1-\\beta_1)\\sum_{i=1}^t\\beta_1^{t-i}=\\mathbb E[g]\\,(1-\\beta_1^{\\,t}). $$So $\\mathbf m_t$ underestimates $\\mathbb E[g]$ by the factor $(1-\\beta_1^t)$; dividing it out removes the bias. Likewise for $\\mathbf v$:\n$$ \\hat{\\mathbf m}_t=\\frac{\\mathbf m_t}{1-\\beta_1^{\\,t}},\\qquad \\hat{\\mathbf v}_t=\\frac{\\mathbf v_t}{1-\\beta_2^{\\,t}}. $$As $t\\to\\infty$ the corrections →1 (vanishing effect); they matter only in early steps.\nThe update $$ \\boxed{\\ \\theta\\leftarrow\\theta-\\eta\\,\\frac{\\hat{\\mathbf m}_t}{\\sqrt{\\hat{\\mathbf v}_t}+\\epsilon}\\ } $$The division by $\\sqrt{\\hat{\\mathbf v}}$ gives each coordinate its own adaptive step: large-gradient directions (big $\\hat v$) are damped; small-gradient directions (small $\\hat v$) are amplified. The effective per-coordinate step is roughly $\\eta\\cdot\\mathrm{sign}$-like and scale-invariant to gradient magnitude — a cheap diagonal preconditioner approximating inverse-curvature scaling, which is why Adam tolerates the ill-conditioning that defeats SGD. Typical: $\\beta_1{=}0.9,\\beta_2{=}0.95\\text{–}0.999,\\epsilon{=}10^{-8}$ ([CONFIG]).\nAdamW decouples weight decay from the adaptive term, applying it as a separate shrinkage:\n$$ \\theta\\leftarrow\\theta-\\eta\\Big(\\frac{\\hat{\\mathbf m}_t}{\\sqrt{\\hat{\\mathbf v}_t}+\\epsilon}+\\lambda\\theta\\Big). $$Decoupling matters because L2 regularization folded into $g$ gets rescaled by $1/\\sqrt{\\hat v}$ (inconsistent across coordinates); AdamW\u0026rsquo;s separate $\\lambda\\theta$ term decays all weights uniformly. AdamW is the de facto standard for LLMs.\nOptimizer memory. Adam stores $\\mathbf m,\\mathbf v\\in\\mathbb R^P$ → $2P$ extra numbers, the dominant reason training needs far more memory than inference. With a master fp32 copy plus fp16/bf16 params and grads, the classic accounting is $\\approx16$ bytes/parameter before activations. This state is [STATE], discarded at deployment.\nAdam decides how the gradient at a given step becomes a step in $\\theta$. It says nothing about how the overall step size should evolve across a run of hundreds of thousands of steps — and at scale that trajectory is not optional.\nFigure 2. The price of two moment buffers.\n4. Learning-rate schedule A constant $\\eta$ is rarely optimal; LLMs use a deliberate trajectory $\\eta_t$ (a planned [CONFIG] curve, not learning):\nWarmup. Ramp $\\eta$ linearly from ~0 over the first $T_w$ steps. Early parameters are random, gradients large and noisy, and Adam\u0026rsquo;s $\\hat{\\mathbf v}$ estimate is still unreliable; large early steps diverge. Warmup eases in. Decay. After the peak, anneal toward ~0, commonly via cosine: $$ \\eta_t=\\eta_{\\min}+\\tfrac12(\\eta_{\\max}-\\eta_{\\min})\\Big(1+\\cos\\tfrac{\\pi(t-T_w)}{T-T_w}\\Big),\\quad t\\gt T_w. $$Large steps early traverse the landscape; small steps late settle into a minimum and reduce gradient-noise-induced wobble. The peak, warmup length, and final value are all [CONFIG] and among the most consequential knobs.\nGradient clipping. To guard against rare loss spikes, the global gradient norm is capped: if $\\|g\\|\\gt c$, rescale $g\\leftarrow c\\,g/\\|g\\|$. A standard stabilizer ([CONFIG], $c\\sim1$).\nThe schedule controls the size of each step; the batch controls how trustworthy the gradient that defines it is. Recall from Lesson 7 that the mini-batch gradient is an unbiased estimate whose variance falls as $1/|B|$ — the next section turns that fact into a sizing decision.\nFigure 3. Warmup, then cosine decay.\n5. Batch size and the noise trade-off Batch size $|B|$ (in tokens) sets the gradient-estimate variance ($\\propto1/|B|$) and hardware utilization. LLMs use very large batches (often $10^6$+ tokens) for stable gradients and parallelism. Two practical mechanisms:\nGradient accumulation: sum gradients over $k$ micro-batches before one update → simulate batch $k|B|$ within fixed memory. Critical batch size: there is a regime beyond which larger batches give diminishing returns (more compute, same progress per token); efficient training targets that knee. There is also an empirical near-linear $\\eta\\!\\propto\\!|B|$ scaling over a useful range. Everything so far makes the descent fast and stable. None of it yet addresses whether the resulting $\\theta$ will work on text it has never seen — the actual goal. That is the job of regularization.\n6. Regularization and generalization A model with $P\\sim$ data size could overfit. Countermeasures ([CONFIG]):\nWeight decay ($\\lambda$, via AdamW): penalizes weight norm, biasing toward smoother functions. Dropout: zero a fraction $p$ of activations during training (off at inference), preventing co-adaptation. Large pretraining runs often use little/no dropout, leaning on data scale instead. Data scale itself: the strongest regularizer — when the token count rivals or exceeds memorization capacity, generalizing is the lower-loss strategy. We now have every per-step and per-run dial. The remaining question is set before training begins: given a fixed compute budget, how large a model and how much data should we even attempt? That is the planning layer.\n7. Scaling laws — the planning layer Empirically, pretraining loss follows a power law in compute, parameters, and data:\n$$ \\mathcal L(N_{\\text{params}},D)\\approx \\mathcal L_\\infty + \\Big(\\tfrac{A}{N_{\\text{params}}}\\Big)^{\\alpha}+\\Big(\\tfrac{B}{D}\\Big)^{\\beta}, $$a smooth, predictable curve down to an irreducible floor $\\mathcal L_\\infty$ (the entropy of language, Lesson 6). The compute-optimal result (Chinchilla) is that for a fixed budget $C\\approx6N_{\\text{params}}D$ (Lesson 1), $N_{\\text{params}}$ and $D$ should scale in roughly equal proportion — many earlier models were under-trained on data for their size. These laws are how labs choose model size, token count, and $\\eta$ before committing a run; they are [CONFIG] planning, informed by fitting the curve on smaller runs. (Appendix A derives the compute-optimal split and the ~20 tokens/param rule in full.)\nThe full state taxonomy Configuration (chosen) Parameters (learned) Optimizer/runtime state (transient) optimizer type, $\\beta_1,\\beta_2,\\epsilon,\\lambda$ $E$, attn $W_{Q,K,V,O}$ $\\mathbf m,\\mathbf v\\in\\mathbb R^P$ $\\eta$ schedule, warmup, clip $c$ MLP $W_1,W_2,b$ cached activations batch size, accumulation $k$ norm $\\gamma,\\beta$ gradients $g$ depth/width/heads/$V$, total steps unembedding $W_U$ loss value Conclusion The bare SGD update fails at scale because a single step size cannot handle heterogeneous gradients and ill-conditioned curvature. Momentum EMAs the gradient to accelerate and denoise; Adam/AdamW adds a second-moment EMA and divides by $\\sqrt{\\hat{\\mathbf v}}$ for a per-coordinate adaptive step (a diagonal preconditioner), with bias correction $\\hat{\\mathbf m}=\\mathbf m/(1-\\beta_1^t)$ derived from the zero-initialized EMA\u0026rsquo;s $(1-\\beta_1^t)$ underestimate, and decoupled weight decay. Its $2P$ moment buffers are the main reason training memory dwarfs inference. The learning rate follows a warmup-then-cosine-decay schedule, guarded by gradient clipping; batch size trades $1/|B|$ variance against cost, with accumulation and a critical-batch knee. Regularization — weight decay, dropout, and above all data scale — buys generalization, and scaling laws ($\\mathcal L\\!\\approx\\!\\mathcal L_\\infty+(A/N)^\\alpha+(B/D)^\\beta$, compute-optimal $N\\!\\propto\\!D$) plan the whole run. Every dial is config; the moments are transient state; only $\\theta$ persists.\nThis completes pretraining. Next: the post-training stages — SFT and RLHF — that turn a next-token predictor into an aligned assistant, and exactly what each updates.\n← Lesson 7 — Backpropagation \u0026amp; Gradient Descent · Course index · Next: Lesson 9 — Fine-Tuning, SFT \u0026amp; RLHF → Appendices: A — Acronyms · B — Formulas · C — Attention Variants The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/tutorials/llm-lessons/lesson-08-optimizers-learning-rate-batches/","summary":"\u003ch1 id=\"lesson-8--optimizers-learning-rate--batches\"\u003eLesson 8 — Optimizers, Learning Rate \u0026amp; Batches\u003c/h1\u003e\n\u003cp\u003e\u003cem\u003eTechnical course on LLM mechanics — Lesson 8 of 10.\u003c/em\u003e\u003c/p\u003e\n\u003chr\u003e\n\u003ch2 id=\"recap-of-lesson-7\"\u003eRecap of Lesson 7\u003c/h2\u003e\n\u003cp\u003eBackprop computes $\\nabla_\\theta\\mathcal L\\in\\mathbb R^P$ in one reverse sweep via VJPs ($\\bar W=\\bar{\\mathbf u}\\mathbf x^\\top$, $\\bar{\\mathbf x}=W^\\top\\bar{\\mathbf u}$), seeded by $\\hat{\\mathbf p}-\\mathbf y$. The bare update is $\\theta\\leftarrow\\theta-\\eta\\nabla_\\theta\\mathcal L$. We estimate the gradient on mini-batches (unbiased, variance $\\propto1/|B|$). Residuals/normalization/scaling keep the Jacobian product conditioned; checkpointing tames memory. This lesson specifies the \u003cstrong\u003eoptimizer\u003c/strong\u003e that maps $g\\mapsto$ update, the \u003cstrong\u003eschedule\u003c/strong\u003e on $\\eta$, and the \u003cstrong\u003eregularizers\u003c/strong\u003e — all \u003cstrong\u003e[CONFIG]\u003c/strong\u003e, plus the optimizer\u0026rsquo;s \u003cstrong\u003e[STATE]\u003c/strong\u003e.\u003c/p\u003e","title":"Optimizers, Learning Rate and Batches"},{"content":"Lesson 8 — Pretraining Dynamics Technical course on encoder LLM mechanics — Lesson 8 of 10.\nRecap of Lesson 7 An encoder has no perplexity (no autoregressive joint), so it is judged by held-out MLM loss, pseudo-perplexity as a weak intrinsic stand-in, and — decisively — downstream GLUE/SQuAD/MTEB scores after fine-tuning. To get a model worth evaluating, we must first pretrain it. This lesson covers how MLM pretraining is actually run, reusing the decoder course\u0026rsquo;s optimization engine wholesale and focusing on the few things that genuinely differ — most importantly the consequences of the 15% signal-density problem from Lesson 5.\n1. The engine is identical There is nothing encoder-specific about the optimization machinery, so we recap it only to set the baseline. The forward/backward/update loop, reverse-mode autodiff, and the optimizer are exactly as in the decoder course. Encoders are trained with AdamW — Adam with decoupled weight decay — whose per-coordinate update is\n$$ \\theta\\leftarrow\\theta-\\eta\\Big(\\frac{\\hat{\\mathbf m}}{\\sqrt{\\hat{\\mathbf v}}+\\epsilon}+\\lambda\\theta\\Big), $$with bias-corrected first/second moments $\\hat{\\mathbf m},\\hat{\\mathbf v}$. The same $2P$ optimizer-state memory, the same mixed-precision and gradient-clipping practices, the same residual-stream gradient flow (Lesson 4) all carry over without modification. [LEARNED] $\\theta$, [CONFIG] $(\\eta,\\lambda,\\beta_1,\\beta_2,\\epsilon,\\text{batch})$, [STATE] $(\\mathbf m,\\mathbf v,\\text{activations})$ — the categories are unchanged.\nContrast with the decoder. At the level of \u0026ldquo;how do gradients become parameter updates,\u0026rdquo; there is no encoder/decoder difference. Everything distinctive about training an encoder lives in the data pipeline and the loss\u0026rsquo;s signal structure, not the optimizer. So this lesson spends its budget there.\n2. Learning-rate warmup, and why post-LN needs it Encoder pretraining uses the standard warmup → decay schedule: ramp $\\eta$ linearly from 0 over the first few thousand steps, then decay (linearly or cosine). Warmup matters more for original BERT than for a typical pre-LN decoder because of the post-LN wiring (Lesson 4): in post-LN, large early updates can destabilize the normalization statistics and blow up training, and warmup is the standard remedy. This is a concrete training consequence of the post-LN choice — pre-LN encoders (ModernBERT) tolerate more aggressive schedules.\nFigure 1. Warmup, then decay — and why post-LN leans on it.\n3. The 15% signal-density problem, quantified This is the central encoder-specific training fact, foreshadowed in Lesson 5. In autoregressive training, a single forward pass over an $n$-token sequence yields $n$ prediction targets — every position contributes a loss term. In MLM, the same-cost forward pass yields only $|\\mathcal M|\\approx0.15\\,n$ loss terms; the other ~85% of positions are fully computed (they participate in attention, consume FLOPs) but produce no direct gradient signal.\nSo the supervision per forward-pass FLOP is roughly $6\\times$ lower for MLM than for autoregressive training. Two consequences:\nMLM is comparatively data-/compute-hungry. To see a target token enough times, you push more tokens through. BERT trained on ~3.3B words (BooksCorpus + Wikipedia); RoBERTa scaled this dramatically (~160GB of text) and trained far longer, and the single biggest source of RoBERTa\u0026rsquo;s gains over BERT was simply more data and more steps — i.e. compensating for low signal density with volume. It motivates denser objectives. ELECTRA\u0026rsquo;s replaced-token detection (Appendix A) scores all positions, recovering the dense-supervision property and reaching BERT-level quality with far less compute. This is a direct attack on the 15% problem. Takeaway. The defining inefficiency of encoder pretraining is not the optimizer or the architecture — it is that the MLM loss illuminates only the masked fraction of each sequence. Scaling data (RoBERTa) or changing the objective (ELECTRA) are the two ways the field bought back the lost signal.\nFigure 2. Full compute per token, a fraction of the supervision.\n4. Masking: static vs. dynamic How the 15% is chosen also affects dynamics. Original BERT applied masking once during preprocessing — each sequence had a fixed mask pattern reused every epoch (static masking). RoBERTa switched to dynamic masking: generate a fresh random mask every time a sequence is seen. Dynamic masking means the model sees each sentence with many different positions hidden over training, which is more data-efficient and slightly improves results — a cheap win that is now standard.\n$$ \\text{static: } \\mathcal M(x)\\text{ fixed} \\qquad\\text{vs.}\\qquad \\text{dynamic: } \\mathcal M(x)\\sim\\text{fresh each epoch}. $$ Contrast with the decoder. A decoder has no masking choice to make — every token is always a target, every epoch. Static-vs-dynamic masking is an encoder-only design axis, arising entirely from the fact that MLM selects which positions to supervise.\n5. Sequence packing and batching Encoders process fixed-length inputs (≤512, Lesson 2), and natural text comes in variable lengths, so the input pipeline must fill the length budget efficiently. Two practices:\nPacking. Concatenate multiple short documents/sentences into one length-$n$ sequence (separated by [SEP]) so that few positions are wasted on [PAD]. Padded positions cost FLOPs but yield nothing, so packing directly improves throughput.\nLarge batches. RoBERTa found that large batch sizes (thousands of sequences), with the learning rate scaled accordingly, improve both optimization stability and final quality. Because MLM\u0026rsquo;s gradient is noisier (signal from only 15% of positions), large batches help average that noise.\nflowchart LR D[\"raw docs (variable length)\"] --\u003e P[\"pack to length n(minimize [PAD])\"] P --\u003e M[\"dynamic 15% mask(fresh per epoch)\"] M --\u003e F[\"forward (whole batch, 1 pass)\"] F --\u003e L[\"loss on masked positions only\"] L --\u003e U[\"AdamW update (warmup→decay)\"] U -.-\u003e|next batch| P Figure 3. The MLM data pipeline, one batch at a time.\n6. Compute accounting The FLOP arithmetic mirrors the decoder\u0026rsquo;s with one twist. A forward pass still costs $\\approx2P$ FLOPs/token and a full training step $\\approx6P$ FLOPs/token — the backward pass is computed over the whole sequence regardless of which positions are scored, because gradients must flow through every token\u0026rsquo;s representation to reach the parameters. So MLM does not save backward-pass compute by scoring few positions; it pays the full $6P$/token while extracting loss from only 15% of tokens. That is precisely why the signal-density problem bites: you pay decoder-scale compute per token but harvest a fraction of the supervision. Memory accounting (≈16 bytes/param for mixed-precision AdamW, plus activations) is identical, and inference (Lesson 3) holds only parameters plus single-pass activations — no KV cache.\nConclusion Encoder pretraining reuses the decoder\u0026rsquo;s optimization engine without change — AdamW, warmup→decay schedules, mixed precision, the $6P$-FLOP step, the same memory accounting. Everything distinctive lives in the data/loss pipeline. The defining issue is signal density: MLM scores only the masked ~15% of positions while paying full per-token compute, so it harvests roughly $6\\times$ less supervision per FLOP than autoregressive training. The field answers this with volume (RoBERTa: vastly more data and steps), dynamic masking (fresh masks each epoch, an encoder-only knob), packing and large batches (fill the length budget, average the noisier gradient), or a denser objective (ELECTRA scores all positions). Post-LN BERT also leans on warmup for stability — a downstream effect of the Lesson 4 wiring choice.\nWith a pretrained encoder in hand, Lesson 9 turns it into a working system: attaching and fine-tuning the four task heads, and contrasting this with the decoder\u0026rsquo;s SFT/RLHF alignment story.\n← Previous: Lesson 7 — Evaluating an Encoder · Course index · Next: Lesson 9 — Fine-Tuning Encoders for Downstream Tasks → Appendices: A — Encoder Family · B — Encoder vs. Decoder Cheat Sheet · C — Acronyms The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/tutorials/encoder-deep-dive/lesson-08-pretraining-dynamics/","summary":"\u003ch1 id=\"lesson-8--pretraining-dynamics\"\u003eLesson 8 — Pretraining Dynamics\u003c/h1\u003e\n\u003cp\u003e\u003cem\u003eTechnical course on encoder LLM mechanics — Lesson 8 of 10.\u003c/em\u003e\u003c/p\u003e\n\u003chr\u003e\n\u003ch2 id=\"recap-of-lesson-7\"\u003eRecap of Lesson 7\u003c/h2\u003e\n\u003cp\u003eAn encoder has no perplexity (no autoregressive joint), so it is judged by held-out MLM loss, pseudo-perplexity as a weak intrinsic stand-in, and — decisively — downstream \u003cstrong\u003eGLUE/SQuAD/MTEB\u003c/strong\u003e scores after fine-tuning. To get a model worth evaluating, we must first pretrain it. This lesson covers how MLM pretraining is actually run, reusing the decoder course\u0026rsquo;s optimization engine wholesale and focusing on the few things that genuinely differ — most importantly the consequences of the 15% signal-density problem from Lesson 5.\u003c/p\u003e","title":"Pretraining Dynamics"},{"content":"Chapter 8 — Sculpting Images from Noise: Diffusion Models Everything in Part II generated language, and all of it generated language the same way: one word at a time, left to right, each word committed before the next. This chapter leaves language behind for images, and in doing so it introduces a completely different philosophy of creation. The systems behind Stable Diffusion, DALL·E, and Midjourney do not build a picture piece by piece in some fixed order. They start with a square of pure visual static and gradually clarify the whole image at once, like a Polaroid developing out of a haze. The method is called diffusion, and once you see its central trick, it is one of the most satisfying ideas in the field.\nThe one idea A diffusion model learns to do one strangely specific thing: look at a grainy, corrupted image and guess what noise is hidden in it. That is the whole skill. It is a noise-spotter, trained to estimate the corruption in a picture at every possible level of messiness — from \u0026ldquo;barely smudged\u0026rdquo; all the way to \u0026ldquo;pure television static.\u0026rdquo;\nOnce you have a reliable noise-spotter, generating an image falls out almost for free. You start with a square of pure random static, ask the model \u0026ldquo;what\u0026rsquo;s the noise here?\u0026rdquo;, subtract a sliver of what it points to, and repeat. Round after round, the static resolves into a sharp, brand-new picture that never existed before. The entire field rests on a single mischievous insight: destroying an image is trivial, while recreating one is the magic — so we only ever learn the hard direction, and we learn it by practicing on the easy one.\nThe asymmetry at the heart of it Picture a sharp photograph onto which you sprinkle sand, one handful at a time. After one handful you can still make out the picture. After a few hundred, it is just structureless sand. Pouring sand requires no skill at all — anyone can do it, and you always know exactly how much you poured. The miracle would be the reverse: removing exactly one handful of sand to reveal the very slightly cleaner image beneath. If you could do that one small reversal reliably, you could begin with a pile of pure sand and, by repeating tiny removals, conjure a photograph out of nothing.\nThis asymmetry is the entire engine. Adding noise is free — a fixed mechanical recipe with no learning involved. Removing noise is the part we train a network to do. And the genius of the approach is that the hard task — \u0026ldquo;turn static into a picture\u0026rdquo; — is broken into hundreds of tiny, almost-easy tasks, each merely \u0026ldquo;make this slightly less noisy.\u0026rdquo; Each small step is so gentle that it becomes simple and stable to learn. This is precisely why diffusion models train reliably and produce such high quality, where earlier image-generators were notoriously temperamental.\nFigure 1. Pouring sand is free; removing one handful is the miracle.\nThe forward process: controlled destruction The \u0026ldquo;adding sand\u0026rdquo; direction is called the forward process. Take a real training image and, step by step, blend in a little random noise, dimming the original a touch each time. Repeat enough times — often a thousand — and the image dissolves completely into pure static, fuzz indistinguishable from any other fuzz.\nTwo things make this elegant. First, it requires no learning whatsoever; the amount of noise to add at each step is decided in advance, like a recipe, and engineers tune that recipe so the model spends its effort where it counts rather than wasting steps on an already-ruined image. Second, and crucially for making the whole thing practical, there is a shortcut. Naively, to know what an image looks like after seven hundred noising steps, you would have to simulate all seven hundred in sequence. But because the noise is added in a particularly well-behaved way, a single calculation can teleport you straight to any noise level you want in one stroke. Want the image as it would look seven hundred steps in? One step, no waiting.\nThat teleport is what makes training cheap, and it has a beautiful consequence: when you jump an image to some noise level, you know exactly what noise you added. That noise is the perfect answer key. The corruption you sprinkle on is simultaneously the very thing you are training the model to predict. You manufacture an unlimited supply of practice problems, each one already graded.\nThe reverse process: the part that\u0026rsquo;s actually learned The \u0026ldquo;removing sand\u0026rdquo; direction is the reverse process, and it is the only part the network learns. Now, the fully honest version of this problem is impossible: hand someone a grainy image and ask \u0026ldquo;what was the original, and which speck should come off first?\u0026rdquo;, and the truthful answer is \u0026ldquo;it depends what the picture was\u0026rdquo; — which is exactly what you don\u0026rsquo;t know at generation time.\nThe clever workaround is that during training we let the model peek at the original, because we are the ones who added the noise in the first place. With the clean image in hand, the question \u0026ldquo;what is the right small step backward?\u0026rdquo; suddenly has a clean, calculable answer. The network practices reproducing that answer without the peek, so that later, at generation time — when no original exists — it can still take the right step. And each backward step is easy to model precisely because each forward step added only a tiny bit of noise; undoing a tiny corruption is a gentle, well-behaved nudge. This is the deep reason the noising must be done in many small steps rather than a few big ones: small steps keep each reversal simple, where big ones would make the reverse a tangled mess with many possible answers.\nThe intimidating theory, demystified This is the point where textbooks reach for fearsome phrases — \u0026ldquo;variational lower bound,\u0026rdquo; a thicket of Greek symbols — and where we will not. The intuition underneath is humble and worth stating once, plainly, so the jargon never frightens you.\nIdeally we would train the model to directly maximize how likely it is to produce real-looking images. But that quantity hides behind a calculation no one can actually perform. So the field borrows an old, quietly audacious trick: if you can\u0026rsquo;t measure the thing you want, build a stand-in you can measure, prove the stand-in never overshoots the real thing, and then push the stand-in up — which drags the real thing up with it. That guaranteed-pessimistic stand-in is the \u0026ldquo;lower bound.\u0026rdquo; And here is the punchline: when you take that intimidating stand-in and grind through the algebra, almost everything cancels out, and the whole towering apparatus collapses into a single line a child could follow — look at a noisy image, guess the noise, measure how wrong you were, done. Square the error, average it, and that is your training signal. Probability theory goes in; plain \u0026ldquo;minimize the error\u0026rdquo; comes out. The researchers even found that a fussy set of weights the theory suggested could simply be thrown away, and the images came out better for it.\nThe network: an hourglass Something has to actually do the noise-spotting, and traditionally that something is a network shaped like an hourglass — wide at top and bottom, pinched in the middle. (Its technical name is a U-Net.) The shape is not arbitrary. The job is \u0026ldquo;image in, image of the noise out\u0026rdquo; — same size in, same size out — and spotting noise well requires seeing at several scales at once, because corruption affects both fine texture (skin grain, fabric weave) and large structure (this region is an eye, that one is sky). A network that only looked locally would miss the forest; one that only looked globally would smear the trees.\nThe hourglass handles both. Going down, the image is progressively squeezed into a small, coarse summary, and at that tiny scale the network can take in the whole picture at a glance and reason about global structure. Going up, it rebuilds the image back to full resolution, restoring fine detail. And dedicated \u0026ldquo;express wires\u0026rdquo; carry high-resolution detail straight across the gap from the down-path to the up-path, so the fine print is not lost in the squeeze. One more thing the network needs to know is how noisy the current image is, since a barely-smudged picture and near-pure static call for very different handling — so the noise level is fed in as a signal broadcast into every layer, quietly whispering \u0026ldquo;you are at level seven hundred of a thousand.\u0026rdquo; (A newer variant replaces the hourglass with a transformer — the same architecture behind language models — but the job, the training, and the noise-spotting role are identical. The architecture is interchangeable; the idea is not.)\nFigure 2. The hourglass: wide, pinched, wide again.\nThe training loop on an index card All this scaffolding exists to make the actual training trivially simple. In plain words, the loop is: grab a real image; pick a random noise level; generate some random noise; use the teleport shortcut to instantly make the noisy version at that level; ask the network to guess the noise; and nudge its knobs toward the right answer. Repeat with the next image. That is the entire training procedure. There is no adversary fighting the model, no running of the slow reverse process, no impossible calculation — just a \u0026ldquo;predict this number\u0026rdquo; problem with an effectively infinite, free supply of practice examples, since every image at every noise level with every random noise pattern is a fresh, pre-graded exercise. Notably, the model never generates a whole image during training. It only ever practices single-step noise-spotting. Generation, the expensive part, is saved entirely for later.\nGeneration: running the noise-spotter backward To actually make a picture, we run the noise-spotter in reverse. Start with a canvas of pure random static. Ask the network what the noise is, and remove a calibrated sliver of it. Add back a tiny bit of fresh randomness. Repeat hundreds or a thousand times, and out of meaningless static a coherent image gradually sharpens into view.\nWhy add fresh randomness back at each step? Because each backward step is meant to draw from a range of plausible images, not snap to the single most likely one. If you always took the most-likely path, the compounding effect over a thousand steps would drag the result toward a bland, over-smoothed average — a mushy blur rather than a crisp, specific picture. The dash of randomness keeps it sharp and varied. The real drawback of this method is speed: because each step needs the output of the previous one, generating a single image means running the network hundreds or a thousand times in a row. Compared with older methods that produced an image in one pass, this is diffusion\u0026rsquo;s main practical weakness.\nThe first big remedy was the realization that you do not actually need all thousand steps. Because the trained network only ever cared about individual noise levels in isolation — not about the specific thousand-rung ladder it happened to be trained on — you are free to descend the ladder in big leaps, skipping most rungs. Twenty or fifty steps can rival a thousand, using the very same trained model, just sampled more efficiently. This faster scheme can also be made fully deterministic, so the same starting static always yields the same final image — which has the lovely bonus of creating a navigable \u0026ldquo;map\u0026rdquo; between random starting points and finished pictures, letting you smoothly blend between images or take a real photo, find its starting point, tweak it, and regenerate an edited version. Step back far enough and the whole staircase of steps reveals itself as a clumsy approximation of a smooth flow, like a current carrying a speck of static steadily toward a finished image — and once you see it that way, you can borrow well-established tools for tracing smooth paths efficiently, taking large accurate strides and bringing good images down to as few as ten or twenty steps.\nFigure 3. Running the noise-spotter backward, hundreds of times.\nSteering it with words A plain diffusion model is a gifted sleepwalker. Left alone it produces plausible faces, landscapes, and rooms — but none of them on request. It has no notion of \u0026ldquo;a red bicycle at dusk.\u0026rdquo; The final pieces of the puzzle are about teaching it to take instruction, and to do so affordably.\nTo follow a prompt, the network is given the text alongside the noisy image. The prompt is first converted into numbers by a text-understanding model, and the image network then consults those words through cross-attention — the very mechanism from Chapter 6 — so that as it works on each region of the image it pulls in the relevant words, the word \u0026ldquo;bicycle\u0026rdquo; tugging on the region where a bicycle should form. The trouble is that a naively prompted model only half-listens, treating the prompt as a gentle suggestion and often wandering off. The elegant fix, and the single most important control in tools like Stable Diffusion, is to train the network to operate both with a prompt and without one, then at generation time compute both — \u0026ldquo;what would I do given the prompt?\u0026rdquo; and \u0026ldquo;what would I do ignoring it?\u0026rdquo; — and exaggerate the difference, pushing the image further in the prompt\u0026rsquo;s direction than it would naturally go. The strength of that exaggeration is the \u0026ldquo;guidance\u0026rdquo; slider users see: turn it up and the model obeys the prompt aggressively, with sharper adherence but less variety; turn it down and it grows more creative but looser.\nOne last innovation is what put high-quality image generation onto ordinary computers rather than data-center clusters. Running diffusion directly on full-resolution pixels — a thousand passes over millions of pixels — is ruinously expensive. So the breakthrough behind Stable Diffusion was to first compress the image into a much smaller \u0026ldquo;essence\u0026rdquo; representation, run the entire diffusion process in that compact space, and only at the very end expand the result back to full pixels. Because the working space can be dozens of times smaller, the cost plummets. This same overall recipe now powers far more than images — audio, video, 3D shapes, even the design of proteins and molecules — with only the data and the network changing, never the core idea.\nDiffusion versus the storytellers It is worth setting diffusion beside the language models of Part II, because they embody opposite philosophies of generation. A language model writes one token at a time, left to right, each word committed before the next is chosen, building an output as an irreversible sequence — like writing a sentence word by word and never going back. A diffusion model works in the opposite spirit: the entire output is present from the start but corrupted, a full canvas of static, and it is refined as a whole, in parallel, every part getting clearer together over many passes — less like writing a sentence and more like a photograph developing.\nThis difference fits their subjects. The left-to-right approach suits language, where order is everything and each word genuinely depends on the ones before it. The refine-the-whole-canvas approach suits images, where every part must be globally coherent at once — the left eye matching the right, the lighting consistent across the frame — which is hard to guarantee if you commit to pixels one at a time in some arbitrary order. Two great families of generators, built on opposite instincts, each matched to the shape of what it creates.\nWhat to carry forward A diffusion model learns to predict the noise in a corrupted image at every level of corruption. Training is \u0026ldquo;add noise once, then learn to estimate it,\u0026rdquo; a simple error-minimizing loop with an endless free supply of pre-graded examples. Generation is running that noise-predictor backward, starting from pure static and gradually removing noise until an image appears. Everything else — the demystified theory, the hourglass network, the fast samplers, the prompt-following and guidance, the compressed working space — is refinement, acceleration, and control layered on top of that one idea. And where language models generate sequentially, diffusion generates the whole thing at once and sharpens it.\nWe have now seen the major architectures, for both language and images. What we have largely deferred is how any of them is actually built and used — the lifecycle that turns a pile of random knobs into a capable, helpful system. That is Part IV, beginning with the question of how a model learns at all, in Chapter 9 .\n← Chapter 7 · The Committee · Contents · Chapter 9 · How a Model Learns → The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/tutorials/inside-the-machines/chapter-08-diffusion-models/","summary":"\u003ch1 id=\"chapter-8--sculpting-images-from-noise-diffusion-models\"\u003eChapter 8 — Sculpting Images from Noise: Diffusion Models\u003c/h1\u003e\n\u003cp\u003eEverything in Part II generated language, and all of it generated language the same way: one word at a time, left to right, each word committed before the next. This chapter leaves language behind for images, and in doing so it introduces a completely different philosophy of creation. The systems behind Stable Diffusion, DALL·E, and Midjourney do not build a picture piece by piece in some fixed order. They start with a square of pure visual static and gradually clarify the \u003cem\u003ewhole image at once\u003c/em\u003e, like a Polaroid developing out of a haze. The method is called \u003cem\u003ediffusion\u003c/em\u003e, and once you see its central trick, it is one of the most satisfying ideas in the field.\u003c/p\u003e","title":"Sculpting Images from Noise: Diffusion Models"},{"content":"Lesson 8 — Training Dynamics, Stability \u0026amp; Scaling Laws Technical course on MoE LLM mechanics — Lesson 8 of 10.\nRecap of Lesson 7 Experts are sharded across devices (expert parallelism), so each MoE layer is bracketed by two all-to-all collectives that route tokens to their experts and back — putting MoE training on a communication-bound critical path and giving load balance a systems meaning (the layer runs at the speed of its busiest expert). With experts, router, balancing (Lesson 5), capacity (Lesson 6), and communication (Lesson 7) all in place, the full training apparatus exists. This lesson studies its behavior: the instabilities it introduces and their fixes, the specialization that does (and doesn\u0026rsquo;t) emerge, and the scaling laws that justify spending all this complexity.\nThe engine underneath is unchanged from the dense course — same cross-entropy, same reverse-mode backprop, same AdamW with a warmup-cosine schedule. We do not re-derive any of it. What changes is the dynamics: a sparse, discretely-routed model trained with auxiliary objectives behaves differently from a smooth dense one, and understanding those differences is what separates an MoE that trains from one that diverges.\n1. Router instability and precision MoE training is empirically less stable than dense training, and the router is the usual culprit. The softmax over experts is sensitive: small changes in $W_g$ can flip top-$k$ selections, abruptly rerouting tokens and causing loss spikes. Two stabilizers matter most.\nThe router z-loss (Lesson 5) is the primary fix: penalizing $(\\log\\sum_i e^{h_i})^2$ keeps router logits from growing large, which keeps the softmax from sharpening into a brittle, near-one-hot regime where tiny perturbations cause large rerouting. Empirically the z-loss removes most MoE-specific divergences.\nRouter precision is the second. Most of the network trains in bf16, but the router softmax should be computed in fp32. The exponentials $e^{h_i}$ and their normalizer are exactly where low-precision rounding corrupts the routing distribution — and a corrupted routing decision is discrete and discontinuous, so a small numerical error can flip a selection and propagate. Keeping the router in fp32 (a tiny fraction of compute, since $W_g$ is tiny) while the experts run in bf16 is standard practice and costs almost nothing. This is a stability concern with no dense analogue: a dense model has no softmax whose rounding can flip a discrete decision mid-forward-pass.\nFigure 1. How a small change in W_g becomes a loss spike.\n2. Expert specialization: what actually emerges It is tempting to imagine experts specialize interpretably — a \u0026ldquo;Python expert,\u0026rdquo; a \u0026ldquo;French expert,\u0026rdquo; a \u0026ldquo;math expert.\u0026rdquo; The empirical reality is more muted. Studies of trained MoEs (Switch, Mixtral) find specialization that is real but weak and mostly low-level: experts tend to fire on syntactic or token-level patterns (punctuation, numbers, certain word classes) more than on clean high-level topics, and the assignment is often distributed — many experts share responsibility for a domain rather than one owning it. Routing is also fairly stable across layers but not cleanly hierarchical. The practical takeaways:\nSpecialization is emergent, never assigned — exactly like the emergent layer roles of the dense course\u0026rsquo;s Lesson 4, but now along the expert axis as well as the depth axis. The model\u0026rsquo;s gains come less from crisp human-legible specialists and more from the sheer extra capacity and the router\u0026rsquo;s freedom to mix experts per token. Fine-grained experts and shared experts (Lesson 2) were introduced precisely to improve specialization — slimmer, more numerous experts give the router more combinations to allocate, and shared experts absorb the common computation so routed experts can specialize on the residual. Figure 2. Specialization is real, weak, and mostly low-level.\n3. The MoE scaling trade-off The scaling-law story is where MoE earns its complexity. Recall the dense Chinchilla picture: loss falls predictably with parameters $N$ and tokens $D$ under a compute budget $C\\approx 6ND$. MoE changes the variables that enter this relationship, because it has two parameter counts (Lesson 1).\nThe empirical finding (Switch Transformer, GLaM, and subsequent MoE scaling studies) is that at a fixed training-compute budget — i.e. fixed $P_{\\text{act}}$ and $D$ — adding experts (raising $P_{\\text{tot}}$) lowers the loss. A sparse model with $P_{\\text{act}}$ activated parameters reaches a lower loss than a dense model with the same $P_{\\text{act}}$, and approaches (without fully matching, per parameter) a much larger dense model — at a fraction of the FLOPs. The rough statements people use:\n$$ \\text{at fixed FLOPs (}P_{\\text{act}},D\\text{ fixed):}\\quad \\mathcal L \\text{ decreases as } E\\ (\\text{hence } P_{\\text{tot}}) \\text{ increases}, $$with diminishing returns: each doubling of experts helps less than the last, and there is a point past which extra experts buy little loss but cost real memory and communication. So MoE does not give \u0026ldquo;free\u0026rdquo; parameters — it gives parameters that are cheap in compute but still cost memory and bandwidth, and the right $E$ is the one that balances the loss gain against those costs for your hardware.\nA useful way to hold the two scaling regimes together:\nScale up by… Dense model MoE model more capacity raise $N$ → more FLOPs and more memory raise $E$ → more memory, ~same FLOPs lower per-token cost at fixed capacity not possible (welded) raise $E$, lower $k$ → fewer FLOPs, same memory metric that improves loss vs. compute loss vs. compute at lower compute, traded for memory Contrast with the dense model. Dense scaling is one-dimensional: spend compute, get capacity and cost together, and read off the loss on the Chinchilla curve. MoE scaling is two-dimensional: you choose how much of your capacity to pay for in compute ($P_{\\text{act}}$) versus in memory ($P_{\\text{tot}}$). The whole reason to accept MoE\u0026rsquo;s training complexity — the balancing loss, the capacity dropping, the all-to-all — is that this second dimension lets you reach a dense-model-like loss at sparse-model-like FLOPs. MoE is, in one line, a way to convert memory budget into model quality without spending compute budget.\nFigure 3. More experts, lower loss, at flat FLOPs.\n4. Recipes that work (and what they fixed) The headline MoE training recipes are worth knowing as concrete instantiations of Lessons 5–8 (see Appendix A for the full field guide):\nSwitch Transformer: $k=1$ routing (the simplest possible), the $\\alpha E\\sum_i f_iP_i$ auxiliary loss, capacity factor with dropping, selective fp32 — showed $k=1$ is enough and made the recipe simple and stable. GLaM: $k=2$, decoder MoE every other layer, demonstrating large quality/FLOP gains at scale. ST-MoE: introduced the router z-loss and careful fine-tuning protocols, the standard stability fixes. Mixtral 8×7B: $E=8,k=2$ MoE in every block, a strong open-weight demonstration that $P_{\\text{tot}}\\approx47\\mathrm B$ / $P_{\\text{act}}\\approx13\\mathrm B$ is competitive with much larger dense models. DeepSeek-MoE / V2 / V3: fine-grained experts + shared experts for better specialization and aux-loss-free bias balancing for purity, scaling to hundreds of experts and $P_{\\text{tot}}$ in the hundreds of billions with small $P_{\\text{act}}$. 5. Learned / config / state Quantity Category Router and expert weights [LEARNED] z-loss weight $\\beta$, router precision (fp32), $E$, $k$, granularity, aux scheme, schedule [CONFIG] Router logits and their magnitude, per-expert load, loss spikes / drop rate (diagnostics) [STATE] Conclusion MoE training runs the same engine as dense training (CE loss, backprop, AdamW) but with different dynamics. It is less stable, chiefly because the router softmax can flip discrete selections — fixed by the z-loss (small logits) and fp32 router precision (no rounding-induced reroutes), neither of which has a dense analogue. Expert specialization emerges but is weak and low-level, which is what motivated fine-grained and shared experts. The payoff is the scaling trade-off: at fixed FLOPs ($P_{\\text{act}},D$ held), raising the expert count $E$ (hence $P_{\\text{tot}}$) lowers the loss with diminishing returns — MoE converts memory budget into quality without spending compute budget, the two-dimensional generalization of the one-dimensional dense Chinchilla picture. The Switch / GLaM / ST-MoE / Mixtral / DeepSeek recipes are concrete points in this design space.\nThe model is now trained. The remaining two lessons take it into deployment: first adapting it to downstream tasks, where sparsity changes how fine-tuning behaves, then serving it, where the memory-vs-compute asymmetry returns as the central fact of inference.\n← Lesson 7 — Distributed Training: Expert Parallelism · Course index · Next: Lesson 9 — Fine-Tuning MoE Models → Appendices: A — MoE Family · B — Dense vs. MoE · C — Acronyms The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/tutorials/moe-lessons/lesson-08-training-dynamics-and-scaling/","summary":"\u003ch1 id=\"lesson-8--training-dynamics-stability--scaling-laws\"\u003eLesson 8 — Training Dynamics, Stability \u0026amp; Scaling Laws\u003c/h1\u003e\n\u003cp\u003e\u003cem\u003eTechnical course on MoE LLM mechanics — Lesson 8 of 10.\u003c/em\u003e\u003c/p\u003e\n\u003chr\u003e\n\u003ch2 id=\"recap-of-lesson-7\"\u003eRecap of Lesson 7\u003c/h2\u003e\n\u003cp\u003eExperts are sharded across devices (\u003cstrong\u003eexpert parallelism\u003c/strong\u003e), so each MoE layer is bracketed by two \u003cstrong\u003eall-to-all\u003c/strong\u003e collectives that route tokens to their experts and back — putting MoE training on a communication-bound critical path and giving load balance a systems meaning (the layer runs at the speed of its busiest expert). With experts, router, balancing (Lesson 5), capacity (Lesson 6), and communication (Lesson 7) all in place, the full training apparatus exists. This lesson studies its \u003cem\u003ebehavior\u003c/em\u003e: the instabilities it introduces and their fixes, the specialization that does (and doesn\u0026rsquo;t) emerge, and the scaling laws that justify spending all this complexity.\u003c/p\u003e","title":"Training Dynamics, Stability and Scaling Laws"},{"content":"Lesson 8 — Training: Teacher Forcing \u0026amp; the Loss Technical course on encoder–decoder transformers — Lesson 8 of 10.\nRecap of Lesson 7 Encoder–decoders are pretrained by denoising: corrupt a document into $\\tilde x$, encode it bidirectionally, and have the decoder autoregressively reconstruct the clean/missing content $y$. T5 uses span corruption (sentinels, short targets); BART mixes five noises and rebuilds the whole document. Either way the loss is cross-entropy over decoder targets with the gradient $\\hat{\\mathbf p}-\\mathbf y$. We now make the training procedure precise — how the decoder is fed and scored — and draw the sharp line between how the model is trained and how it later generates.\n1. The tension: autoregression vs. parallel training The decoder is autoregressive: $P_\\theta(y_t\\mid y_{\\lt t},x)$ depends on the previous output tokens. At inference those previous tokens are the model\u0026rsquo;s own (yet-to-be-generated) outputs, so generation is inherently sequential. But at training time we already know the entire correct output $y$ — it is the target. So we can sidestep the sequentiality entirely: feed the decoder the ground-truth previous tokens at every position, rather than its own predictions. This is teacher forcing, and it is what lets us train all $m$ output positions in a single parallel forward pass instead of $m$ sequential ones.\nTeacher forcing. During training, the decoder\u0026rsquo;s input at position $t$ is the true token $y_{t-1}$, not the model\u0026rsquo;s prediction $\\hat y_{t-1}$. The model is always conditioned on a correct prefix, so every position can be computed at once.\n2. Shifted inputs and the causal mask, working together Mechanically, teacher forcing is implemented by feeding the decoder the target sequence shifted right by one, prepended with a start token (\u0026lt;bos\u0026gt; / T5\u0026rsquo;s \u0026lt;pad\u0026gt;-as-start):\n$$ \\text{decoder input} = (\\langle\\text{bos}\\rangle,\\,y_1,\\,y_2,\\,\\dots,\\,y_{m-1}), \\qquad \\text{targets} = (y_1,\\,y_2,\\,\\dots,\\,y_m). $$Position $t$ of the decoder receives $y_{t-1}$ as input and is asked to predict $y_t$. The decoder\u0026rsquo;s causal self-attention mask (Lesson 3) is what makes this safe in parallel: even though the entire shifted target is fed in at once, position $t$ can only attend to inputs $\\le t$, so it physically cannot see $y_t$ or later — no label leakage. The combination is the whole trick:\nteacher forcing supplies the correct prefix (so we don\u0026rsquo;t need to generate), and the causal mask prevents any position from peeking at its own answer (so parallel ≠ cheating). Meanwhile cross-attention attends over the full $H_{\\text{enc}}$ with no causal mask — correct, because the input is fully known at every step regardless of the output position.\nflowchart LR enc[\"Encoder (once): x̃ → H_enc\"] -.cross-attn.-\u003e dec tgt[\"shifted target: ⟨bos⟩ y1 y2 … y_{m-1}\"] --\u003e dec[\"Decoder (parallel, causal mask)\"] dec --\u003e logits[\"logits z_1 … z_m\"] logits --\u003e ce[\"CE vs. y_1 … y_m (all positions at once)\"] Figure 1. The shift by one, position by position.\n3. The loss, summed over positions With the shifted setup, one forward pass yields $m$ logit vectors $\\{\\mathbf z_t\\}$, hence $m$ predictions, and the loss averages cross-entropy over them:\n$$ \\mathcal L(\\theta)=-\\frac{1}{m}\\sum_{t=1}^{m}\\log P_\\theta(y_t\\mid y_{\\lt t},x) =-\\frac{1}{m}\\sum_{t=1}^{m}\\log\\big[\\mathrm{softmax}(\\mathbf z_t)\\big]_{y_t}. $$Just as in the decoder-only case, the per-logit gradient is the clean residual\n$$ \\frac{\\partial\\mathcal L}{\\partial\\mathbf z_t}=\\hat{\\mathbf p}_t-\\mathbf y_t\\quad(\\text{predicted}-\\text{one-hot target}), $$and padding positions in the target (from batching variable-length outputs) are excluded from the average via a target mask — they contribute no loss and no gradient. For fine-tuning on supervised pairs (Lesson 10) the same loss applies with $(x,y)$ the real source/target instead of a corrupted/clean pair; only the data changes.\n4. The gradient\u0026rsquo;s journey: through cross-attention into the encoder This is the part unique to two-stack training, and Lesson 4 §5 promised the detail. The loss lives only at the decoder\u0026rsquo;s output head, yet the encoder must learn. Backpropagation routes the signal like this:\n$$ \\hat{\\mathbf p}_t-\\mathbf y_t \\ \\xrightarrow{W_U}\\ \\frac{\\partial\\mathcal L}{\\partial H_{\\text{dec}}} \\ \\xrightarrow{\\text{dec blocks}}\\ \\frac{\\partial\\mathcal L}{\\partial(\\text{cross-attn }K,V)} \\ \\xrightarrow{W_K^c,\\,W_V^c}\\ \\frac{\\partial\\mathcal L}{\\partial H_{\\text{enc}}} \\ \\xrightarrow{\\text{enc blocks}}\\ \\frac{\\partial\\mathcal L}{\\partial(\\text{encoder weights})}. $$Every decoder block contributes a term to $\\partial\\mathcal L/\\partial H_{\\text{enc}}$ through its cross-attention $K,V$ projections; these sum, and the total flows backward through the whole encoder. So the two stacks are optimized jointly, end to end in one backward pass — the encoder\u0026rsquo;s gradient is entirely mediated by cross-attention. This is the precise sense in which \u0026ldquo;cross-attention is the only new mechanism\u0026rdquo; governs training as well as the forward pass: it is the sole channel through which the encoder discovers what representations the decoder needs. (Memory cost mirrors this: training caches activations for both stacks, so an encoder–decoder\u0026rsquo;s activation memory exceeds a decoder-only model\u0026rsquo;s of comparable decoder size.)\nFigure 2. How a loss at the decoder\u0026rsquo;s head trains the encoder.\n5. Training vs. inference: the gap that matters We can now state the two phases side by side, because their difference drives the next lesson.\nTraining (this lesson) Inference (Lesson 9) Decoder input at $t$ ground-truth $y_{t-1}$ (teacher forcing) model\u0026rsquo;s own previous output $\\hat y_{t-1}$ Positions computed all $m$ at once (parallel) one at a time (sequential) Encoder runs once per example once per example Loss / gradient yes (writes $\\theta$) none ($\\theta$ frozen) Output prefix is always correct possibly already wrong The last row is the seed of exposure bias: the model is only ever trained on correct prefixes, but at inference it must condition on its own prefixes, which may contain errors it never saw during training. A single early mistake can push the decoder into a region of prefix-space it was never optimized on, and errors compound. Teacher forcing buys parallel, stable training at the cost of this train/test mismatch. Mitigations (scheduled sampling, sequence-level objectives, or just large-scale pretraining making the model robust) exist, but the standard recipe accepts the gap — and inference-time techniques like beam search (Lesson 9) partly compensate by not committing greedily to a single prefix.\nLearned / config / state Quantity Category All weights updated by the backward pass [LEARNED] Teacher-forcing policy, shift-by-one convention, start token, LR schedule, batch size [CONFIG] Target padding mask (data-dependent) [CONFIG]/[STATE] Logits $\\mathbf z_t$, cached activations (both stacks), gradients, optimizer moments [STATE] Figure 3. Teacher forcing, and the gap it leaves behind.\nConclusion Training uses teacher forcing: the decoder is fed the ground-truth output shifted right by one, so every output position conditions on a correct prefix and all $m$ positions can be scored in a single parallel forward pass. The decoder\u0026rsquo;s causal mask keeps that parallelism honest (no position sees its own target), while cross-attention reads the full encoder memory unmasked. The loss is averaged cross-entropy with the clean $\\hat{\\mathbf p}_t-\\mathbf y_t$ logit gradient, and that gradient flows from the decoder head through cross-attention into the encoder, training both stacks jointly — the encoder\u0026rsquo;s only learning signal. The price is exposure bias: trained only on correct prefixes, the model must later condition on its own, possibly-wrong outputs.\nThat mismatch is exactly what the inference phase must manage. Lesson 9 turns the frozen, trained model loose: encode once, decode many, with two distinct KV caches, beam search and length normalization for quality, and coverage and copy mechanisms for the failure modes specific to seq2seq generation.\n← Lesson 7 — Pretraining Objectives · Course index · Next: Lesson 9 — Inference → Appendices: A — Formulas · B — Architecture Comparison · C — Acronyms The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/tutorials/encoder-decoder-lessons/lesson-08-training-teacher-forcing/","summary":"\u003ch1 id=\"lesson-8--training-teacher-forcing--the-loss\"\u003eLesson 8 — Training: Teacher Forcing \u0026amp; the Loss\u003c/h1\u003e\n\u003cp\u003e\u003cem\u003eTechnical course on encoder–decoder transformers — Lesson 8 of 10.\u003c/em\u003e\u003c/p\u003e\n\u003chr\u003e\n\u003ch2 id=\"recap-of-lesson-7\"\u003eRecap of Lesson 7\u003c/h2\u003e\n\u003cp\u003eEncoder–decoders are pretrained by \u003cstrong\u003edenoising\u003c/strong\u003e: corrupt a document into $\\tilde x$, encode it bidirectionally, and have the decoder autoregressively reconstruct the clean/missing content $y$. T5 uses span corruption (sentinels, short targets); BART mixes five noises and rebuilds the whole document. Either way the loss is cross-entropy over decoder targets with the gradient $\\hat{\\mathbf p}-\\mathbf y$. We now make the \u003cem\u003etraining procedure\u003c/em\u003e precise — how the decoder is fed and scored — and draw the sharp line between how the model is trained and how it later generates.\u003c/p\u003e","title":"Training: Teacher Forcing \u0026 the Loss"},{"content":"Lesson 9 — Faster \u0026amp; Deterministic Sampling (DDIM) and the Score View Recap of Lesson 8. Inference runs the reverse chain: start at $x_T\\sim\\mathcal N(0,I)$ and iterate $x_{t-1}=\\frac{1}{\\sqrt{\\alpha_t}}(x_t-\\frac{1-\\alpha_t}{\\sqrt{1-\\bar\\alpha_t}}\\epsilon_\\theta) + \\sigma_t z$ down to $x_0$. The deterministic part denoises; $\\sigma_t z$ keeps samples diverse. An equivalent \u0026ldquo;predict $\\hat x_0$, re-noise to $t{-}1$\u0026rdquo; view clarifies each step. The cost — ~$T=1000$ sequential network calls — is diffusion\u0026rsquo;s main weakness. We now make sampling fast and connect to score-based generative modeling.\nA thousand sequential steps to make one picture is the kind of bill that limits what a method is allowed to become. This lesson is the escape, and it arrives in two acts. First the practical trick: the same trained network can descend the mountain in twenty strides instead of a thousand, and can be made to walk a single deterministic trail rather than a drunkard\u0026rsquo;s stagger. Then the deeper turn — the recognition that the trail was never a staircase at all but a smooth current, a differential equation, of which everything we have built is just one discretization among many.\nOverview DDPM\u0026rsquo;s slowness is structural: its reverse chain is Markov, so $x_{t-1}$ requires $x_t$ for all $T$ steps. The DDIM insight (Song et al., 2021a) is that the training objective $L_\\text{simple}$ depends only on the marginals $q(x_t\\mid x_0)$, not on the joint forward process. One can therefore construct an entire non-Markovian family of forward (and hence reverse) processes that all share those marginals — so the same trained $\\epsilon_\\theta$ is valid for every member. Members of this family permit (a) reverse updates that jump across arbitrary timestep subsequences and (b) a tunable injected-noise level $\\sigma_t$ that can be sent to zero, making sampling deterministic.\nA second, deeper lens identifies $\\epsilon_\\theta$ with a scaled estimate of the score $\\nabla_{x_t}\\log q_t(x_t)$. This recasts diffusion as the numerical solution of a stochastic differential equation, or of its deterministic probability-flow ODE — exposing the machinery of stochastic calculus and high-order ODE solvers that bring sampling to $\\sim10$–$20$ steps. These two lenses are the same object viewed twice: deterministic DDIM is the first-order discretization of the probability-flow ODE.\nHeads-up — a deliberate change in rigor. The DDIM half of this lesson (above) stays at the careful, fully-derived level of Lessons 2–5. The score / SDE half (below) is the one genuine difficulty spike in the course: it draws on stochastic calculus (Itô diffusions, Wiener processes, the Fokker–Planck equation) that we have not built up and will not. By design we therefore state the key results and sketch the intuition rather than prove them from first principles — the proofs live in the cited papers (Anderson 1982; Song et al. 2021b). You can read this section for the picture — \u0026ldquo;$\\epsilon_\\theta$ is a score; generation is integrating a reverse-time SDE or its ODE twin\u0026rdquo; — and lose nothing needed for Lesson 10 or for using the models. Treat the \u0026ldquo;Derivation (idea)\u0026rdquo; and \u0026ldquo;Proof sketch\u0026rdquo; boxes as motivation, not as the load-bearing inductive proofs of the earlier lessons.\nDDIM: non-Markovian, optionally deterministic The escape begins with a question that sounds almost like cheating: what if the chain we trained was never the only chain consistent with what the network actually learned?\nThe non-Markovian forward family Fix the DDPM marginals $q(x_t\\mid x_0)=\\mathcal N(\\sqrt{\\bar\\alpha_t}\\,x_0,(1-\\bar\\alpha_t)I)$. Define a family indexed by a vector $\\sigma=(\\sigma_1,\\dots,\\sigma_T)$ of inference reverse-conditionals\n$$ q_\\sigma(x_{t-1}\\mid x_t,x_0)=\\mathcal N\\!\\Big(\\sqrt{\\bar\\alpha_{t-1}}\\,x_0+\\sqrt{1-\\bar\\alpha_{t-1}-\\sigma_t^2}\\;\\frac{x_t-\\sqrt{\\bar\\alpha_t}\\,x_0}{\\sqrt{1-\\bar\\alpha_t}},\\;\\sigma_t^2 I\\Big). $$The mean is engineered so that the family is marginal-preserving: by induction $q_\\sigma(x_t\\mid x_0)=\\mathcal N(\\sqrt{\\bar\\alpha_t}\\,x_0,(1-\\bar\\alpha_t)I)$ for every choice of $\\sigma$.\nDerivation (marginal consistency). Assume $x_t=\\sqrt{\\bar\\alpha_t}\\,x_0+\\sqrt{1-\\bar\\alpha_t}\\,\\epsilon$ with $\\epsilon\\sim\\mathcal N(0,I)$. Substituting into the mean above, the data term contributes $\\sqrt{\\bar\\alpha_{t-1}}\\,x_0$ and the second term contributes $\\sqrt{1-\\bar\\alpha_{t-1}-\\sigma_t^2}\\,\\epsilon$. Adding the noise $\\sigma_t z$ ($z\\perp\\epsilon$), the conditional variance of $x_{t-1}$ given $x_0$ is $(1-\\bar\\alpha_{t-1}-\\sigma_t^2)+\\sigma_t^2=1-\\bar\\alpha_{t-1}$, and the mean is $\\sqrt{\\bar\\alpha_{t-1}}\\,x_0$. Hence $x_{t-1}=\\sqrt{\\bar\\alpha_{t-1}}\\,x_0+\\sqrt{1-\\bar\\alpha_{t-1}}\\,\\epsilon'$, $\\epsilon'\\sim\\mathcal N(0,I)$ — exactly $q(x_{t-1}\\mid x_0)$. The marginals are independent of $\\sigma$. $\\square$\nBecause the marginals match, the variational objective reduces (Song et al., 2021a, Thm. 1) to the same per-step denoising regression $L_\\text{simple}$ used in Lesson 5. The trained $\\epsilon_\\theta$ is therefore reused unchanged across the whole family.\nThe DDIM reverse update At sampling time $x_0$ is unknown; we substitute the model\u0026rsquo;s estimate $\\hat x_0=(x_t-\\sqrt{1-\\bar\\alpha_t}\\,\\epsilon_\\theta(x_t,t))/\\sqrt{\\bar\\alpha_t}$ (Tweedie/posterior-mean, below) into the conditional mean, giving:\n$$ \\boxed{\\, x_{t-1} = \\sqrt{\\bar\\alpha_{t-1}}\\;\\underbrace{\\left(\\frac{x_t - \\sqrt{1-\\bar\\alpha_t}\\,\\epsilon_\\theta(x_t,t)}{\\sqrt{\\bar\\alpha_t}}\\right)}_{\\hat x_0 \\text{ (predicted clean image)}} \\;+\\; \\underbrace{\\sqrt{1-\\bar\\alpha_{t-1}-\\sigma_t^2}\\;\\epsilon_\\theta(x_t,t)}_{\\text{direction pointing to } x_t} \\;+\\; \\sigma_t z \\,} $$Read it as three pieces: predict the clean image $\\hat x_0$, scale it to level $t-1$, add a deterministic \u0026ldquo;direction\u0026rdquo; term built from the same predicted noise, and optionally add stochastic noise $\\sigma_t z$.\nRecovering the two extremes $\\sigma_t=\\sqrt{\\tilde\\beta_t}$ recovers DDPM. Setting the injected variance to the true posterior variance $\\tilde\\beta_t=\\frac{1-\\bar\\alpha_{t-1}}{1-\\bar\\alpha_t}\\beta_t$ makes $q_\\sigma(x_{t-1}\\mid x_t,x_0)$ coincide with the DDPM posterior $q(x_{t-1}\\mid x_t,x_0)$ from Lesson 3, and the update reduces algebraically to the Lesson 8 sampler. The process becomes Markov again. $\\sigma_t=0$ gives deterministic DDIM. All randomness vanishes after $x_T$; the map $x_T\\mapsto x_0$ is a deterministic function. The update becomes $$ x_{t-1}=\\sqrt{\\bar\\alpha_{t-1}}\\,\\hat x_0+\\sqrt{1-\\bar\\alpha_{t-1}}\\;\\epsilon_\\theta(x_t,t), $$ which we will recognize below as the first-order integrator of the probability-flow ODE. Remark (interpolating regimes). A common parameterization is $\\sigma_t=\\eta\\,\\sqrt{\\tilde\\beta_t}$ with $\\eta\\in[0,1]$: $\\eta=1$ is DDPM, $\\eta=0$ is deterministic DDIM, and intermediate $\\eta$ trades sample diversity against few-step fidelity. The choice is orthogonal to training.\nTwo big payoffs Fewer steps. Because the update references $\\bar\\alpha$ at arbitrary levels, it runs on a subsequence $\\tau_1\\lt \\dots\\lt \\tau_S$ of $\\{1,\\dots,T\\}$ with $S\\ll T$. Each accelerated step maps $x_{\\tau_i}\\to x_{\\tau_{i-1}}$ using $\\bar\\alpha_{\\tau_i},\\bar\\alpha_{\\tau_{i-1}}$. With $\\eta=0$, 20–50 steps often rival 1000 DDPM steps. A meaningful latent space. Deterministic DDIM, being an ODE discretization, is approximately invertible: running the update forward (the DDIM inversion, $x_t\\to x_{t+1}$ with the same $\\epsilon_\\theta$) recovers a near-unique $x_T$. This makes $x_T\\leftrightarrow x_0$ a near-bijection, enabling semantic interpolation (slerp two $x_T$\u0026rsquo;s, decode, get a smooth morph) and real-image editing (invert a real image to $x_T$, edit, regenerate) — both impossible with stochastic DDPM. Remark (inversion fidelity). DDIM inversion is exact only in the continuous limit; at finite step counts and with strong classifier-free guidance (Lesson 10) the local linearization error accumulates, and reconstruction drifts. Null-text inversion and exact-inversion solvers exist to mitigate this.\nFigure 1. One knob, two familiar samplers, and two payoffs.\nThe score-based / SDE perspective (This is the stated-results-and-intuition section flagged in the overview; stochastic-calculus background is assumed but not required to follow the picture.)\nThe unifying lens (Song \u0026amp; Ermon, 2019; Song et al., 2021b). Define the score of a distribution as the gradient of its log-density w.r.t. the input:\n$$ s(x) := \\nabla_x \\log p(x). $$Tweedie\u0026rsquo;s formula and the score identity Theorem (Tweedie\u0026rsquo;s formula; Robbins, Efron). Let $x_0\\sim p$ and let $x_t=\\sqrt{\\bar\\alpha_t}\\,x_0+\\sqrt{1-\\bar\\alpha_t}\\,\\epsilon$, $\\epsilon\\sim\\mathcal N(0,I)$, so the noisy marginal is $q_t(x_t)=\\int \\mathcal N(x_t;\\sqrt{\\bar\\alpha_t}x_0,(1-\\bar\\alpha_t)I)\\,p(x_0)\\,dx_0$. Then the posterior mean of the clean scaled signal satisfies $$ \\gt \\mathbb E[\\sqrt{\\bar\\alpha_t}\\,x_0\\mid x_t]=x_t+(1-\\bar\\alpha_t)\\,\\nabla_{x_t}\\log q_t(x_t). \\gt $$ Proof sketch. For the Gaussian likelihood $q_t(x_t\\mid x_0)=\\mathcal N(\\sqrt{\\bar\\alpha_t}x_0,(1-\\bar\\alpha_t)I)$, differentiate under the integral: $\\nabla_{x_t}q_t(x_t)=\\int \\frac{\\sqrt{\\bar\\alpha_t}x_0-x_t}{1-\\bar\\alpha_t}\\,q_t(x_t\\mid x_0)p(x_0)\\,dx_0$. Dividing by $q_t(x_t)$ gives $\\nabla_{x_t}\\log q_t(x_t)=\\frac{1}{1-\\bar\\alpha_t}\\big(\\mathbb E[\\sqrt{\\bar\\alpha_t}x_0\\mid x_t]-x_t\\big)$. Rearranging yields the stated formula. $\\square$\nNow connect to the noise. Since $x_t=\\sqrt{\\bar\\alpha_t}x_0+\\sqrt{1-\\bar\\alpha_t}\\,\\epsilon$, the posterior mean of $\\sqrt{\\bar\\alpha_t}x_0$ equals $x_t-\\sqrt{1-\\bar\\alpha_t}\\,\\mathbb E[\\epsilon\\mid x_t]$. Substituting into Tweedie:\n$$ x_t-\\sqrt{1-\\bar\\alpha_t}\\,\\mathbb E[\\epsilon\\mid x_t]=x_t+(1-\\bar\\alpha_t)\\nabla_{x_t}\\log q_t(x_t) \\;\\Longrightarrow\\; \\nabla_{x_t}\\log q_t(x_t)=-\\frac{\\mathbb E[\\epsilon\\mid x_t]}{\\sqrt{1-\\bar\\alpha_t}}. $$The denoising-score-matching optimum is $\\epsilon_\\theta(x_t,t)=\\mathbb E[\\epsilon\\mid x_t]$ (the MSE-minimizer is the conditional mean), so:\n$$ \\boxed{\\ \\nabla_{x_t}\\log q_t(x_t) \\approx -\\frac{\\epsilon_\\theta(x_t,t)}{\\sqrt{1-\\bar\\alpha_t}}\\ } $$So training $\\epsilon_\\theta$ to predict noise is, up to the known scaling $-1/\\sqrt{1-\\bar\\alpha_t}$, training a model of the score at every noise level. The simple loss of Lesson 5 is weighted denoising score matching in disguise. Equivalently, the posterior-mean estimate $\\hat x_0$ used in the DDIM update is the Tweedie estimator.\nWith a score model one can also generate by annealed Langevin dynamics — stepping along the score plus noise:\n$$ x \\leftarrow x + \\tfrac{\\delta}{2}\\,\\nabla_x\\log p(x) + \\sqrt{\\delta}\\,z, $$a stochastic walk toward high-density regions; this is the original NCSN sampler (Song \u0026amp; Ermon, 2019).\nContinuous time: the VP-SDE Take $T\\to\\infty$ so the discrete chain becomes a continuous-time stochastic differential equation (SDE). The DDPM discretization corresponds to the variance-preserving (VP) SDE (Song et al., 2021b). Writing $\\beta(t)$ for the continuous-time noise rate,\n$$ \\text{forward (VP-SDE):}\\quad dx = -\\tfrac12\\,\\beta(t)\\,x\\,dt + \\sqrt{\\beta(t)}\\,dw, $$with $w$ a standard Wiener process; here $f(x,t)=-\\tfrac12\\beta(t)x$ and $g(t)=\\sqrt{\\beta(t)}$. Its transition kernel is Gaussian with mean-scale $\\exp(-\\tfrac12\\int_0^t\\beta(s)ds)$ and matches the discrete $\\bar\\alpha_t$ in the limit, recovering $q_t(x_t\\mid x_0)=\\mathcal N(\\sqrt{\\bar\\alpha_t}x_0,(1-\\bar\\alpha_t)I)$.\nTheorem (Anderson, 1982; reverse-time SDE). A diffusion $dx=f(x,t)dt+g(t)dw$ run forward from $t=0$ to $T$ admits a time-reversal that is itself a diffusion: $$ \\gt dx=\\big[f(x,t)-g(t)^2\\,\\nabla_x\\log q_t(x)\\big]dt+g(t)\\,d\\bar w, \\gt $$ where $\\bar w$ is a Wiener process in reverse time and $q_t$ is the forward marginal at time $t$. The reverse drift depends on the data only through the score $\\nabla_x\\log q_t$.\nSubstituting the learned score $-\\epsilon_\\theta/\\sqrt{1-\\bar\\alpha_t}$ and integrating this reverse SDE from $t=T$ to $0$ is generation. The DDPM ancestral sampler of Lesson 8 is the Euler–Maruyama discretization of the VP reverse SDE.\nThe probability-flow ODE — the fast lane Every diffusion SDE has an associated deterministic probability-flow ODE whose trajectories have the same marginal densities $q_t$ at every time (Song et al., 2021b, Prop. 1, via the Fokker–Planck equation):\n$$ \\frac{dx}{dt} = f(x,t) - \\tfrac{1}{2}g(t)^2\\,\\nabla_x\\log q_t(x). $$ Derivation (idea). The forward Fokker–Planck equation $\\partial_t q_t=-\\nabla\\!\\cdot(f q_t)+\\tfrac12 g^2\\nabla^2 q_t$ can be rewritten as a continuity equation $\\partial_t q_t=-\\nabla\\!\\cdot(\\tilde f q_t)$ with effective drift $\\tilde f=f-\\tfrac12 g^2\\nabla\\log q_t$ and no diffusion term. Any deterministic flow with velocity $\\tilde f$ transports the same density family, hence shares the marginals. $\\square$\nBecause it is an ODE, one can apply mature numerical integrators — Heun/RK methods, DEIS, and DPM-Solver (Lu et al., 2022) exploit the semilinear structure (the exact solution of the linear part plus a learned-term integral) to reach high quality in as few as 10–20 steps. The EDM framework (Karras et al., 2022) further clarifies preconditioning, noise schedules, and second-order correction.\nRemark (DDIM = first-order ODE discretization). Substituting the score identity into the VP probability-flow ODE and integrating one step with the exponential-integrator (change of variables to $\\lambda_t=\\log(\\sqrt{\\bar\\alpha_t}/\\sqrt{1-\\bar\\alpha_t})$, the log-SNR) yields exactly the deterministic ($\\sigma_t=0$) DDIM update. DDIM is therefore the first-order member of the DPM-Solver family, and higher-order solvers are its principled generalization.\nFigure 2. From a noise predictor to a differential equation.\nPredicting $v$ instead of $\\epsilon$ One practical footnote before we leave the chapter — the regression target that quietly powers most fast and high-resolution models.\nA practical reparameterization used in fast/distilled and high-resolution models: predict $v := \\sqrt{\\bar\\alpha_t}\\,\\epsilon - \\sqrt{1-\\bar\\alpha_t}\\,x_0$ (Salimans \u0026amp; Ho, 2022). The $v$-parameterization keeps the regression target well-scaled across all noise levels — at $t\\to T$ where $\\epsilon$-prediction is fine but $x_0$-prediction degenerates, and at $t\\to 0$ where the reverse holds — and is the natural target on the velocity field of the probability-flow ODE. It improves stability at high resolution and is the standard target for progressive distillation, which halves the step count repeatedly to compress 1000 steps into a handful.\nFigure 3. Velocity along the diffusion arc.\nCode: a DDIM sampler import torch @torch.no_grad() def ddim_sample(model, shape, alphabar, timesteps, eta=0.0, device=\u0026#34;cuda\u0026#34;): \u0026#34;\u0026#34;\u0026#34; model: eps_theta(x_t, t) -\u0026gt; noise estimate, same shape as x_t shape: (B, C, H, W) alphabar: tensor of cumulative products bar_alpha_t, indexed by global t in [0, T-1] timesteps: decreasing subsequence of ints, e.g. list(range(999, -1, -20)) eta: 0.0 -\u0026gt; deterministic DDIM; 1.0 -\u0026gt; DDPM-equivalent stochasticity returns: x_0 estimate, shape == shape \u0026#34;\u0026#34;\u0026#34; x = torch.randn(shape, device=device) # x_T ~ N(0, I), (B,C,H,W) for i, t in enumerate(timesteps): t_next = timesteps[i + 1] if i + 1 \u0026lt; len(timesteps) else -1 ab_t = alphabar[t] # scalar bar_alpha_t ab_prev = alphabar[t_next] if t_next \u0026gt;= 0 else torch.tensor(1.0, device=device) t_batch = torch.full((shape[0],), t, device=device, dtype=torch.long) # (B,) eps = model(x, t_batch) # (B,C,H,W) # Tweedie / predicted clean image hat_x0 (B,C,H,W) x0_pred = (x - (1 - ab_t).sqrt() * eps) / ab_t.sqrt() x0_pred = x0_pred.clamp(-1, 1) # optional stabilization # sigma_t = eta * sqrt(tilde_beta_t); tilde_beta_t = (1-ab_prev)/(1-ab_t) * (1 - ab_t/ab_prev) sigma = eta * (((1 - ab_prev) / (1 - ab_t)) * (1 - ab_t / ab_prev)).clamp(min=0).sqrt() # direction term uses the SAME eps; (1 - ab_prev - sigma^2) \u0026gt;= 0 by construction dir_xt = (1 - ab_prev - sigma**2).clamp(min=0).sqrt() * eps noise = sigma * torch.randn_like(x) if eta \u0026gt; 0 else 0.0 x = ab_prev.sqrt() * x0_pred + dir_xt + noise # x_{t_next}, (B,C,H,W) return x Numerical notes. (i) alphabar indexing must use the global timestep, not the loop index, so accelerated subsequences read the correct $\\bar\\alpha$. (ii) Clamping $\\hat x_0$ to the data range stabilizes few-step sampling and is the simple precursor of dynamic thresholding (Lesson 10). (iii) The clamp(min=0) guards against tiny negative radicands from float error near $t\\approx T$. (iv) For classifier-free guidance, replace eps with the guided estimate before computing x0_pred (Lesson 10). (v) Use float32 for alphabar; accumulating $\\bar\\alpha_t$ in float16 loses precision at small $\\beta_t$.\nExercises 1. Marginal consistency. In the DDIM family, with $x_t=\\sqrt{\\bar\\alpha_t}\\,x_0+\\sqrt{1-\\bar\\alpha_t}\\,\\epsilon$, show that the conditional variance of $x_{t-1}$ given $x_0$ equals $1-\\bar\\alpha_{t-1}$ (so the marginal matches $q(x_{t-1}\\mid x_0)$ for any $\\sigma_t$).\nThe mean\u0026rsquo;s \u0026ldquo;direction\u0026rdquo; term contributes $\\sqrt{1-\\bar\\alpha_{t-1}-\\sigma_t^2}\\,\\epsilon$ (variance $1-\\bar\\alpha_{t-1}-\\sigma_t^2$) and the added stochastic term contributes $\\sigma_t z$ with $z\\perp\\epsilon$ (variance $\\sigma_t^2$). Being independent, the variances add: $(1-\\bar\\alpha_{t-1}-\\sigma_t^2)+\\sigma_t^2=1-\\bar\\alpha_{t-1}$. The mean is $\\sqrt{\\bar\\alpha_{t-1}}\\,x_0$, so $x_{t-1}\\sim\\mathcal N(\\sqrt{\\bar\\alpha_{t-1}}x_0,(1-\\bar\\alpha_{t-1})I)=q(x_{t-1}\\mid x_0)$, independent of $\\sigma_t$.\n2. Recovering the two extremes. Show that $\\sigma_t=0$ gives the deterministic DDIM update, and explain why $\\sigma_t=\\sqrt{\\tilde\\beta_t}$ makes the process Markov again (DDPM).\nSetting $\\sigma_t=0$ drops the $\\sigma_t z$ term and makes $\\sqrt{1-\\bar\\alpha_{t-1}-\\sigma_t^2}=\\sqrt{1-\\bar\\alpha_{t-1}}$, leaving $x_{t-1}=\\sqrt{\\bar\\alpha_{t-1}}\\,\\hat x_0+\\sqrt{1-\\bar\\alpha_{t-1}}\\,\\epsilon_\\theta$ — a deterministic map. Setting $\\sigma_t=\\sqrt{\\tilde\\beta_t}$ injects exactly the true posterior variance of Lesson 3, so $q_\\sigma(x_{t-1}\\mid x_t,x_0)$ coincides with the DDPM posterior $q(x_{t-1}\\mid x_t,x_0)$ and the update reduces to the Lesson 8 ancestral sampler — the chain is Markov again.\n3. The score–noise identity. Combine Tweedie\u0026rsquo;s formula $\\mathbb E[\\sqrt{\\bar\\alpha_t}x_0\\mid x_t]=x_t+(1-\\bar\\alpha_t)\\nabla_{x_t}\\log q_t(x_t)$ with $\\sqrt{\\bar\\alpha_t}x_0=x_t-\\sqrt{1-\\bar\\alpha_t}\\,\\epsilon$ to derive $\\nabla_{x_t}\\log q_t(x_t)=-\\mathbb E[\\epsilon\\mid x_t]/\\sqrt{1-\\bar\\alpha_t}$, and hence the $\\epsilon_\\theta\\leftrightarrow$ score relation.\nTaking conditional expectations, $\\mathbb E[\\sqrt{\\bar\\alpha_t}x_0\\mid x_t]=x_t-\\sqrt{1-\\bar\\alpha_t}\\,\\mathbb E[\\epsilon\\mid x_t]$. Equating with Tweedie:\n$$ x_t-\\sqrt{1-\\bar\\alpha_t}\\,\\mathbb E[\\epsilon\\mid x_t]=x_t+(1-\\bar\\alpha_t)\\nabla_{x_t}\\log q_t(x_t) \\;\\Rightarrow\\; \\nabla_{x_t}\\log q_t(x_t)=-\\frac{\\mathbb E[\\epsilon\\mid x_t]}{\\sqrt{1-\\bar\\alpha_t}}. $$Since the MSE-optimal predictor is the conditional mean, $\\epsilon_\\theta(x_t,t)\\approx\\mathbb E[\\epsilon\\mid x_t]$, giving $\\nabla_{x_t}\\log q_t(x_t)\\approx-\\epsilon_\\theta(x_t,t)/\\sqrt{1-\\bar\\alpha_t}$: predicting noise is (scaled) score estimation.\n4. DDIM as a first-order solver (conceptual). In one or two sentences, explain in what precise sense deterministic DDIM is the first-order discretization of the probability-flow ODE.\nSubstituting the score identity into the VP probability-flow ODE and integrating one step with an exponential integrator (changing variables to the log-SNR $\\lambda_t=\\log(\\sqrt{\\bar\\alpha_t}/\\sqrt{1-\\bar\\alpha_t})$) reproduces exactly the $\\sigma_t=0$ DDIM update. DDIM is thus the first-order member of the DPM-Solver family; higher-order solvers (Heun, DPM-Solver-2/3) are principled generalizations that take larger, more accurate steps.\nConclusion DDIM arises from a non-Markovian forward family with marginals matched to $q(x_t\\mid x_0)$, so the trained $\\epsilon_\\theta$ transfers unchanged; choosing $\\sigma_t=\\sqrt{\\tilde\\beta_t}$ recovers DDPM and $\\sigma_t=0$ gives a deterministic, timestep-skipping map (20–50 steps rivaling 1000) with a near-invertible latent space for interpolation and editing. The score view, grounded in Tweedie\u0026rsquo;s formula, shows $\\epsilon_\\theta$ is a scaled estimate of $\\nabla_{x_t}\\log q_t(x_t)$, recasting generation as integrating Anderson\u0026rsquo;s reverse-time SDE — or its marginal-equivalent probability-flow ODE, which Heun/DEIS/DPM-Solver crack in $\\sim10$–$20$ steps, with deterministic DDIM as the first-order case. The $v$-parameterization keeps targets well-scaled for distillation and high resolution. We can now train and sample efficiently. Lesson 10 adds the final, practical layer: conditioning and guidance (steering generation with text or labels) and latent diffusion (how Stable Diffusion makes it all affordable).\nFast to train, fast to sample, and revealed as the discretization of a continuous flow — the chain has given up its last secret. Only one capability is still missing: the ability to ask for what we want. Lesson 10 supplies it, and with it the modern text-to-image stack.\nReferences J. Song, C. Meng, S. Ermon. Denoising Diffusion Implicit Models (DDIM). ICLR 2021a. Y. Song, S. Ermon. Generative Modeling by Estimating Gradients of the Data Distribution (NCSN). NeurIPS 2019. Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, B. Poole. Score-Based Generative Modeling through Stochastic Differential Equations. ICLR 2021b. B. D. O. Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Applications, 1982. B. Efron. Tweedie\u0026rsquo;s Formula and Selection Bias. JASA 2011 (after H. Robbins, 1956). C. Lu, Y. Zhou, F. Bao, J. Chen, C. Li, J. Zhu. DPM-Solver. NeurIPS 2022. T. Karras, M. Aittala, T. Aila, S. Laine. Elucidating the Design Space of Diffusion-Based Generative Models (EDM). NeurIPS 2022. T. Salimans, J. Ho. Progressive Distillation for Fast Sampling of Diffusion Models ($v$-prediction). ICLR 2022. ← Previous: Lesson 8 — DDPM Sampling · Course Index · Next: Lesson 10 — Conditioning \u0026amp; Guidance → · Appendix A: Acronyms · Appendix B: Formulas The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/tutorials/diffusion-lessons/lesson-09-ddim-and-score/","summary":"\u003ch1 id=\"lesson-9--faster--deterministic-sampling-ddim-and-the-score-view\"\u003eLesson 9 — Faster \u0026amp; Deterministic Sampling (DDIM) and the Score View\u003c/h1\u003e\n\u003cblockquote\u003e\n\u003cp\u003e\u003cstrong\u003eRecap of Lesson 8.\u003c/strong\u003e Inference runs the reverse chain: start at $x_T\\sim\\mathcal N(0,I)$ and iterate $x_{t-1}=\\frac{1}{\\sqrt{\\alpha_t}}(x_t-\\frac{1-\\alpha_t}{\\sqrt{1-\\bar\\alpha_t}}\\epsilon_\\theta) + \\sigma_t z$ down to $x_0$. The deterministic part denoises; $\\sigma_t z$ keeps samples diverse. An equivalent \u0026ldquo;predict $\\hat x_0$, re-noise to $t{-}1$\u0026rdquo; view clarifies each step. The cost — ~$T=1000$ sequential network calls — is diffusion\u0026rsquo;s main weakness. We now make sampling fast and connect to score-based generative modeling.\u003c/p\u003e","title":"DDIM and Score-Based Models"},{"content":"Lesson 9 — Fine-Tuning Encoders for Downstream Tasks Technical course on encoder LLM mechanics — Lesson 9 of 10.\nRecap of Lesson 8 MLM pretraining reuses the decoder\u0026rsquo;s AdamW engine; its distinctive cost is signal density — only the masked ~15% of positions yield loss — answered by more data (RoBERTa), dynamic masking, packing, and large batches. Pretraining produces a model whose representations are good but generic: it can reconstruct masked tokens, but it does not yet do your task. This lesson turns the pretrained encoder into a working classifier/tagger/extractor by attaching and training the heads of Lesson 6 — and contrasts this with how a decoder is adapted (SFT, RLHF, or just prompting).\n1. The fine-tuning recipe, in one paragraph Take the pretrained stack, discard the MLM head, attach a small task head (Lesson 6), and continue training on a labeled dataset with the task\u0026rsquo;s loss. Almost always you fine-tune all of $\\theta$ — the whole encoder plus the new head — end to end. The head is randomly initialized; the encoder starts from pretrained weights. Because the encoder already encodes language, fine-tuning needs little data ($10^3$–$10^5$ labeled examples), few epochs (2–4), and a small learning rate ($\\sim2$–$5\\times10^{-5}$) to adapt without destroying (\u0026ldquo;catastrophically forgetting\u0026rdquo;) the pretrained representations. This transfer-learning move — pretrain once on unlabeled text, fine-tune cheaply per task — is the entire reason BERT reshaped NLP.\nWhat updates. All of $\\theta$ (encoder) plus the head, by the usual AdamW rule. The pretrained weights are the initialization; fine-tuning moves them to a task-specific region of parameter space.\n2. The four task patterns Each task pairs an input format (Lesson 2) with a head (Lesson 6) and a loss.\nSequence classification — sentiment, topic, intent. Input: [CLS] sentence [SEP]. Pool to $\\mathbf c$, classify, cross-entropy loss against the label:\n$$ \\mathcal L=-\\log\\mathrm{softmax}(W_{\\text{cls}}\\mathbf c+\\mathbf b)_{y}. $$Sentence-pair classification — natural-language inference (entailment/contradiction/neutral), paraphrase, semantic similarity. Input: [CLS] A [SEP] B [SEP] with segment embeddings distinguishing A and B. Same pooled head; for similarity regression output a scalar with MSE loss. This is where the segment embedding (Lesson 2) and [CLS] pooling earn their keep.\nToken classification — NER, POS tagging. Input: [CLS] w₁ … w_k [SEP]. Per-token head, cross-entropy summed over real tokens:\n$$ \\mathcal L=-\\sum_{i\\in\\mathcal R}\\log\\mathrm{softmax}(W_{\\text{tok}}H^{(N)}_{i,:}+\\mathbf b)_{y_i}. $$A subtlety: WordPiece splits words into sub-tokens (play, ##ing), so the label is typically attached to the first sub-token of each word and the continuation pieces are ignored in the loss.\nSpan extraction (QA) — SQuAD-style. Input: [CLS] question [SEP] passage [SEP]. Span head predicts start/end distributions over passage positions; loss is the sum of two cross-entropies against the gold start and end:\n$$ \\mathcal L=-\\tfrac12\\big(\\log P_{\\text{start}}(i^\\star)+\\log P_{\\text{end}}(j^\\star)\\big). $$flowchart TD PT[\"pretrained encoder (drop MLM head)\"] --\u003e A[\"+ pooled classifier→ sentiment / NLI / paraphrase\"] PT --\u003e B[\"+ per-token classifier→ NER / POS\"] PT --\u003e C[\"+ span head→ extractive QA\"] A --\u003e FT[\"fine-tune ALL θ + headsmall LR, 2–4 epochs\"] B --\u003e FT C --\u003e FT Figure 1. One backbone, four task patterns.\n3. Parameter-efficient fine-tuning Full fine-tuning produces a complete copy of the model per task — wasteful when serving many tasks. The same parameter-efficient methods used for decoders apply unchanged to encoders:\nLoRA. Freeze the pretrained $W_0$ and learn a low-rank correction $W=W_0+\\frac{\\alpha}{r}BA$ with $A\\in\\mathbb R^{r\\times k}$, $B\\in\\mathbb R^{d\\times r}$, $r\\ll\\min(d,k)$. Only $BA$ trains; one frozen backbone serves many tasks by swapping tiny adapters.\nAdapters. Insert small bottleneck MLPs between layers and train only those. Feature-extraction / linear probing. Freeze the encoder entirely and train only the head — cheapest, slightly weaker, useful when labeled data is scarce or you need one backbone for everything.\nContrast with the decoder. LoRA and adapters are identical tools here — this part of the story does not differ. What differs is the surrounding philosophy (next section).\nFigure 2. Three ways to spend fewer trainable parameters.\n4. The big contrast: fine-tuning heads vs. aligning a generator This is the sharpest divergence in the course. A decoder, after pretraining, is adapted by:\nSFT — continue the same next-token loss on curated (prompt, response) demonstrations, masked to response tokens, to elicit instruction-following. RLHF / DPO — optimize against a preference signal so the generator\u0026rsquo;s outputs are ranked the way humans prefer. Often nothing at all — a capable decoder does many tasks zero-/few-shot from a prompt, with frozen weights. An encoder is adapted by attaching a typed head and supervised fine-tuning on labeled examples. There is no RLHF, no preference optimization, no prompting-to-elicit — those are all about steering generation, and an encoder does not generate. Conversely, the decoder rarely \u0026ldquo;attaches a head\u0026rdquo;; it reuses its one vocabulary head and changes behavior through data and prompts.\nDimension Decoder adaptation Encoder adaptation Mechanism SFT (same LM loss), RLHF/DPO, or prompting attach typed head + supervised FT Per-task labels few or none (few-shot) required (labeled dataset) Output generated text structured label / tags / span Architecture change none (reuse LM head) new head per task Alignment to preferences central (RLHF/DPO) not applicable (no generation) Contrast with the decoder. The decoder\u0026rsquo;s adaptation problem is \u0026ldquo;make the generator helpful, honest, and harmless\u0026rdquo; — a behavioral problem solved with preference learning. The encoder\u0026rsquo;s adaptation problem is \u0026ldquo;map these representations to this label space\u0026rdquo; — a supervised-learning problem solved with a head and gradient descent. They barely overlap, and that is the cleanest summary of the two model families\u0026rsquo; roles.\nFigure 3. Aligning a generator, against mapping representations.\n5. Practical pitfalls Two encoder-specific gotchas worth naming. First, instability on small datasets: fine-tuning BERT on tiny tasks (a few hundred examples) is famously sensitive to random seed and learning rate; standard mitigations are more warmup, smaller LR, and multiple seeds. Second, the [MASK] mismatch (Lesson 5): pretraining used [MASK], but fine-tuning never does — the 80/10/10 recipe was designed to make this gap small, and in practice fine-tuning closes it, but it is the reason naive \u0026ldquo;prompt BERT by inserting [MASK]\u0026rdquo; tricks (pattern-exploiting training) need care.\nConclusion Fine-tuning an encoder is transfer learning at its cleanest: discard the MLM head, attach a small typed head matched to the task (pooled classifier, per-token classifier, or span head), and continue training all of $\\theta$ plus the head on a labeled dataset with a small learning rate for a few epochs. The four canonical patterns — sequence classification, sentence-pair classification, token classification, and span QA — each pair an input format with a head and a loss. Parameter-efficient methods (LoRA, adapters, linear probing) transfer over from decoders unchanged. The deep contrast is in philosophy: a decoder is aligned (SFT/RLHF/DPO) or prompted to steer its generation and often needs no per-task training, while an encoder is supervised-fine-tuned with a typed head to map representations to a label space and always needs labels. Generation-steering vs. representation-mapping is the two families\u0026rsquo; division of labor.\nThe final lesson zooms out: where encoders actually win today — embeddings, retrieval, RAG — and the model family from RoBERTa through ModernBERT, closing with the practical question of when to reach for an encoder versus a decoder.\n← Previous: Lesson 8 — Pretraining Dynamics · Course index · Next: Lesson 10 — Encoders in Practice \u0026amp; the Modern Landscape → Appendices: A — Encoder Family · B — Encoder vs. Decoder Cheat Sheet · C — Acronyms The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/tutorials/encoder-deep-dive/lesson-09-finetuning-downstream/","summary":"\u003ch1 id=\"lesson-9--fine-tuning-encoders-for-downstream-tasks\"\u003eLesson 9 — Fine-Tuning Encoders for Downstream Tasks\u003c/h1\u003e\n\u003cp\u003e\u003cem\u003eTechnical course on encoder LLM mechanics — Lesson 9 of 10.\u003c/em\u003e\u003c/p\u003e\n\u003chr\u003e\n\u003ch2 id=\"recap-of-lesson-8\"\u003eRecap of Lesson 8\u003c/h2\u003e\n\u003cp\u003eMLM pretraining reuses the decoder\u0026rsquo;s AdamW engine; its distinctive cost is \u003cstrong\u003esignal density\u003c/strong\u003e — only the masked ~15% of positions yield loss — answered by more data (RoBERTa), dynamic masking, packing, and large batches. Pretraining produces a model whose representations are good but generic: it can reconstruct masked tokens, but it does not yet do \u003cem\u003eyour\u003c/em\u003e task. This lesson turns the pretrained encoder into a working classifier/tagger/extractor by attaching and training the heads of Lesson 6 — and contrasts this with how a decoder is adapted (SFT, RLHF, or just prompting).\u003c/p\u003e","title":"Fine-Tuning Downstream"},{"content":"Lesson 9 — Fine-Tuning MoE Models Technical course on MoE LLM mechanics — Lesson 9 of 10.\nRecap of Lesson 8 MoE training runs the same engine as dense training but with shakier dynamics — the router softmax can flip discrete selections, fixed by the z-loss and fp32 router precision — and its payoff is the scaling trade-off: raising the expert count lowers loss at fixed FLOPs, converting memory budget into quality. Specialization emerges weakly. With a pretrained sparse base model in hand, this lesson adapts it: supervised fine-tuning and alignment, where sparsity changes how adaptation behaves and introduces failure modes the dense recipe never sees.\nPretraining built one frozen $\\theta$. The companion dense course\u0026rsquo;s Lesson 9 reshaped it into an assistant via SFT and RLHF/DPO — the same gradient descent, restricted to demonstration and preference data. All of that machinery applies unchanged to an MoE: the loss masking, LoRA, the Bradley–Terry reward, PPO, DPO are agnostic to whether the FFN is dense or sparse. What is not unchanged is the interaction between fine-tuning\u0026rsquo;s small, narrow datasets and the model\u0026rsquo;s sparse, conditionally-trained parameters. That interaction is this lesson.\n1. Why sparse models overfit fine-tuning data The signature fine-tuning problem of MoE is overfitting, and it follows directly from sparsity plus the gradient asymmetry of Lesson 4.\nTwo compounding effects:\nHuge $P_{\\text{tot}}$, small dataset. An MoE has the parameter count of a very large model ($P_{\\text{tot}}$), but fine-tuning datasets are small (thousands to millions of examples) compared to pretraining (trillions of tokens). A high-capacity model on a small dataset overfits — and the MoE\u0026rsquo;s capacity is large by $P_{\\text{tot}}$, not the smaller $P_{\\text{act}}$ that governed its compute. Sparse, uneven gradient coverage. On a small fine-tuning set, not every expert receives enough tokens to be updated well (Lesson 4: an expert only learns from tokens routed to it). The experts that do get the narrow task\u0026rsquo;s tokens are updated aggressively and can overfit, while others drift or stay stale. The router, meanwhile, may overfit its routing decisions to the small distribution. Empirically, ST-MoE documented this clearly: large sparse models that beat dense models at pretraining could underperform them after naive fine-tuning, purely because of overfitting. The fixes are about regularizing and restricting the adaptation.\nFigure 1. Why the sparse model turns up sooner.\n2. Strategies that work Stronger regularization / smaller learning rates. Because the effective capacity is large, MoE fine-tuning uses more dropout and/or lower learning rates than the dense equivalent.\nExpert dropout. A targeted regularizer: apply a higher dropout rate inside the expert FFNs than in the rest of the network. This directly counteracts the experts\u0026rsquo; tendency to memorize the narrow fine-tuning distribution while leaving attention and the router less perturbed.\nSelective / dense fine-tuning. Rather than updating all of $\\theta$, fine-tune a subset:\nupdate only the non-expert parameters (attention, norms, embeddings) — surprisingly effective, since much task adaptation lives outside the FFNs, and it sidesteps the sparse-coverage problem entirely; or update only the experts; or use parameter-efficient methods (LoRA, from the dense course) on the attention and/or expert matrices, which adds a tiny low-rank $\\frac{\\alpha}{r}BA$ update and leaves the bulk of $\\theta$ frozen — especially attractive here because the bulk is enormous. Freeze the router. A choice unique to MoE: keep $W_g$ frozen during fine-tuning so the routing pattern learned in pretraining is preserved, and adapt only the experts (or non-expert weights). This prevents the router from overfitting its decisions to the small dataset and collapsing the carefully-balanced load from pretraining. The alternative — a trainable router — is more flexible but risks re-introducing the imbalance of Lesson 5 on a dataset too small to rebalance against. Whether the router is frozen is a [CONFIG] decision with no dense counterpart (a dense model has no router to freeze).\nKeep a balancing pressure (or don\u0026rsquo;t). If the router is trained during fine-tuning, the auxiliary balancing loss (Lesson 5) is usually kept on, though often with a smaller weight, to stop the narrow data from re-collapsing the router. If the router is frozen, the balancing loss is unnecessary.\nFigure 2. Regularize and restrict — the MoE fine-tuning toolkit.\n3. Alignment (SFT / RLHF / DPO) on an MoE The alignment pipeline itself is unchanged — the same response-masked SFT loss, the same Bradley–Terry reward model, the same PPO clipped objective or DPO loss from the companion course. An MoE is a drop-in policy/reward backbone. The MoE-specific considerations are exactly the ones above carried into alignment:\nSFT and DPO datasets are small, so the overfitting and router-freezing considerations of §1–§2 apply directly. The reward model, if itself an MoE, inherits the same fine-tuning cautions. Inference-time behavior of the aligned model is governed by sampling knobs as usual (next lesson) — alignment does not touch the routing mechanism, only the weights. Contrast with the dense model. Dense fine-tuning has one model and one straightforward concern (don\u0026rsquo;t overfit; use LoRA for efficiency). MoE fine-tuning has three extra decisions, all stemming from sparsity: (1) the model overfits more because effective capacity is $P_{\\text{tot}}$, not $P_{\\text{act}}$; (2) gradient coverage of the experts is uneven on small data, so which-parameters-to-update (experts vs. non-experts vs. LoRA) is a live choice; and (3) there is a router to freeze or not, a knob that simply does not exist in a dense model. The alignment math is identical; the regularization and which-parameters strategy is where MoE diverges.\nFigure 3. The alignment math is identical; the regularization is not.\n4. Learned / config / state Quantity Category Fine-tuned weights (experts, non-experts, router, and/or LoRA adapters) [LEARNED] What to update (experts / non-experts / LoRA), router frozen?, expert-dropout rate, LR, aux-loss weight during FT [CONFIG] Per-batch routing on the FT distribution, drop rate, expert-load shift [STATE] Conclusion Adapting an MoE uses the same SFT/RLHF/DPO machinery as a dense model, but sparsity changes how it behaves. The defining issue is overfitting: effective capacity is the large $P_{\\text{tot}}$, fine-tuning data is small, and gradient coverage across experts is uneven (an expert only learns from its routed tokens). The MoE-specific toolkit is regularization-and-restriction: stronger/expert-targeted dropout, lower learning rates, selective fine-tuning (non-experts only, experts only, or LoRA), and the uniquely-MoE choice of freezing the router to protect the pretrained routing balance (with the auxiliary loss kept on only if the router is trainable). The alignment objectives are identical to the dense case; the divergence is entirely in regularization and in which parameters you choose to move.\nThe model is now trained and adapted. The final lesson serves it — and the memory-vs-compute asymmetry introduced in Lesson 1 returns as the single most important fact about MoE inference.\n← Lesson 8 — Training Dynamics, Stability \u0026amp; Scaling Laws · Course index · Next: Lesson 10 — Inference in Depth → Appendices: A — MoE Family · B — Dense vs. MoE · C — Acronyms The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/tutorials/moe-lessons/lesson-09-finetuning-moe/","summary":"\u003ch1 id=\"lesson-9--fine-tuning-moe-models\"\u003eLesson 9 — Fine-Tuning MoE Models\u003c/h1\u003e\n\u003cp\u003e\u003cem\u003eTechnical course on MoE LLM mechanics — Lesson 9 of 10.\u003c/em\u003e\u003c/p\u003e\n\u003chr\u003e\n\u003ch2 id=\"recap-of-lesson-8\"\u003eRecap of Lesson 8\u003c/h2\u003e\n\u003cp\u003eMoE training runs the same engine as dense training but with shakier dynamics — the router softmax can flip discrete selections, fixed by the z-loss and fp32 router precision — and its payoff is the scaling trade-off: raising the expert count lowers loss at fixed FLOPs, converting memory budget into quality. Specialization emerges weakly. With a pretrained sparse base model in hand, this lesson adapts it: supervised fine-tuning and alignment, where sparsity changes \u003cem\u003ehow\u003c/em\u003e adaptation behaves and introduces failure modes the dense recipe never sees.\u003c/p\u003e","title":"Fine-Tuning MoE Models"},{"content":"Lesson 9 — Fine-Tuning, SFT \u0026amp; RLHF Technical course on LLM mechanics — Lesson 9 of 10.\nRecap of Lesson 8 Adam/AdamW maps the mini-batch gradient to a per-coordinate adaptive step via two EMAs ($\\mathbf m,\\mathbf v$), bias-corrected and divided as $\\hat{\\mathbf m}/(\\sqrt{\\hat{\\mathbf v}}+\\epsilon)$, with decoupled weight decay; its $2P$ buffers dominate training memory. A warmup→cosine-decay $\\eta$ schedule with gradient clipping, large batches, and data-scale regularization, planned via scaling laws, completes pretraining. The product is a base model: an excellent estimator of $P_\\theta(x_t\\mid x_{\\lt t})$ over web text — but not an aligned assistant. This lesson covers the post-training stages and, per the course theme, exactly what each updates. The throughline: every stage is the same gradient descent of Lessons 6–8; only the data and the loss\u0026rsquo;s source change.\nThe engine room (Lessons 7–8) gave us a way to turn any differentiable loss into a trained $\\theta$. Pretraining pointed that engine at one objective — predict the next web token — and got back something fluent but uncooperative. This lesson is where we change what we point the engine at, not how it runs: the mechanism is identical gradient descent throughout, and the entire story of alignment is a story about swapping the data and the source of the loss signal. Keep one eye on the \u0026ldquo;what updates\u0026rdquo; column; that is the spine of the lesson.\n1. Why pretraining is not enough The pretraining objective is \u0026ldquo;model the distribution of internet text.\u0026rdquo; That yields knowledge and fluency but optimizes for plausible continuation, not helpful response. A base model prompted with a question may continue with more questions (a common web pattern). The desired behaviors — instruction-following, honesty, refusal of harmful requests, formatting — are latent capabilities that must be elicited and reweighted, not learned from scratch. Hence two further stages. The first, and simplest, just shows the model what a good answer looks like.\n2. Supervised fine-tuning (SFT) Assemble a curated dataset of (prompt $x$, ideal response $y$) demonstrations and continue training with the same cross-entropy next-token loss — but mask the loss to the response tokens only:\n$$ \\mathcal L_{\\text{SFT}}(\\theta)=-\\sum_{t}\\,\\mathbb 1[t\\in\\text{response}]\\;\\log P_\\theta\\big(y_t\\mid x,\\,y_{\\lt t}\\big). $$The indicator zeroes the gradient on prompt tokens: we want the model to learn to generate good responses conditioned on prompts, not to model the prompts themselves. Differences from pretraining are quantitative, not mechanical — far less data ($10^3$–$10^6$ examples), lower $\\eta$, few epochs — chosen to shift behavior without catastrophic forgetting of pretrained knowledge.\nWhat updates: all of $\\theta$, by the usual rule. SFT moves $\\theta$ to a region that reliably follows instructions.\nLoRA: parameter-efficient fine-tuning Updating all $P$ parameters (and storing $2P$ Adam moments) is costly. LoRA freezes the pretrained $W_0$ and learns a low-rank correction:\n$$ W=W_0+\\Delta W,\\qquad \\Delta W=\\frac{\\alpha}{r}\\,BA,\\quad A\\in\\mathbb R^{r\\times k},\\ B\\in\\mathbb R^{d\\times r},\\ r\\ll\\min(d,k). $$Only $A,B$ are [LEARNED] (with $A$ Gaussian-initialized, $B=0$ so training starts at the base model); $W_0$ is [CONFIG]-frozen. The scaling $\\alpha/r$ keeps the update magnitude roughly $r$-independent. Trainable parameters drop from $dk$ to $r(d+k)$ — often $\\lt 1\\%$ — and at inference $\\Delta W$ can be merged into $W_0$ (zero added latency). The update math is identical gradient descent on a far smaller parameter set. (QLoRA additionally quantizes the frozen $W_0$ to 4-bit to fit larger models on one GPU.)\nSFT can only teach what a human was willing to write out as an ideal answer. For the many prompts where no single answer is clearly best, demonstrations run out — but judgments do not. The next two stages learn from comparison instead of imitation, starting by turning human preference into a number the model can be trained against.\nFigure 1. Full fine-tuning, LoRA, and what each has to store.\n3. RLHF, stage A: the reward model For many prompts there is no single demonstrable \u0026ldquo;ideal\u0026rdquo; response, but humans can compare two candidates more reliably than they can author the best one. Collect preference data: for prompt $x$, responses $(y_w,y_l)$ with $y_w$ preferred (\u0026ldquo;won\u0026rdquo;) over $y_l$ (\u0026ldquo;lost\u0026rdquo;). Fit a scalar reward model $r_\\phi(x,y)$ (a copy of the LM with the unembedding replaced by a scalar head) under the Bradley–Terry preference model, where $P(y_w\\succ y_l)=\\sigma\\!\\big(r_\\phi(x,y_w)-r_\\phi(x,y_l)\\big)$. The negative log-likelihood loss:\n$$ \\mathcal L_{\\text{RM}}(\\phi)=-\\,\\mathbb E_{(x,y_w,y_l)}\\Big[\\log\\sigma\\big(r_\\phi(x,y_w)-r_\\phi(x,y_l)\\big)\\Big]. $$Minimizing it (gradient descent on $\\phi$) makes the reward gap track human preference. The reward model is a learned, queryable surrogate for human judgment that can be evaluated millions of times automatically. What updates: $\\phi$ — a separate model, not the policy.\nWith a differentiable surrogate for human judgment in hand, we can finally optimize the model itself toward \u0026ldquo;preferred\u0026rdquo; rather than merely \u0026ldquo;plausible.\u0026rdquo; But the target is now a learned, imperfect function, which forces a more careful update than next-token imitation.\n4. RLHF, stage B: policy optimization (PPO) Now optimize the LM policy $\\pi_\\theta$ to produce high-reward responses. There is no fixed target token to imitate; the loss comes from the reward of sampled responses. The KL-regularized objective:\n$$ \\max_\\theta\\ \\mathbb E_{x\\sim\\mathcal D,\\,y\\sim\\pi_\\theta(\\cdot\\mid x)}\\Big[\\,r_\\phi(x,y)\\;-\\;\\beta\\,\\log\\frac{\\pi_\\theta(y\\mid x)}{\\pi_{\\text{ref}}(y\\mid x)}\\Big]. $$The first term rewards human-preferred outputs; the second is a per-token KL penalty anchoring $\\pi_\\theta$ to the frozen SFT reference $\\pi_{\\text{ref}}$. The KL term is essential: without it the policy drifts into degenerate text that over-optimizes (games) the imperfect reward model — a Goodhart failure. $\\beta$ ([CONFIG]) sets the leash length.\nThis is optimized with PPO, a policy-gradient method. Two quantities drive its update. The advantage $\\hat A_t$ answers \u0026ldquo;was the token actually generated here better or worse than what this policy would typically produce in this state?\u0026rdquo; — it is the reward attributed to that step minus a learned baseline (a value estimate of the state), so $\\hat A_t\\gt 0$ means \u0026ldquo;better than expected, do more of this\u0026rdquo; and $\\hat A_t\\lt 0$ means \u0026ldquo;worse, do less.\u0026rdquo; Subtracting the baseline keeps the signal centred near zero, which sharply reduces the variance of the gradient. The second quantity is the probability ratio $\\rho_t(\\theta)=\\pi_\\theta(a_t\\mid s_t)/\\pi_{\\theta_{\\text{old}}}(a_t\\mid s_t)$, measuring how much the updated policy has changed the likelihood of that token versus the policy that generated it. PPO maximizes the clipped surrogate:\n$$ \\mathcal L^{\\text{CLIP}}(\\theta)=\\mathbb E_t\\Big[\\min\\big(\\rho_t\\hat A_t,\\ \\mathrm{clip}(\\rho_t,1-\\epsilon,1+\\epsilon)\\,\\hat A_t\\big)\\Big]. $$The clip prevents any single update from moving the policy too far (a trust region), which stabilizes training on the noisy, non-stationary reward signal. What updates: $\\theta$ (the policy), still by gradient ascent on this objective; the reward model $\\phi$ and reference $\\pi_{\\text{ref}}$ are frozen during this stage. This stage is most responsible for the assistant \u0026ldquo;feel\u0026rdquo;: tone, refusals, calibrated hedging.\nPPO works, but it is heavy: a separate reward model, a sampling loop, and a delicate trust region to tune. A natural question is whether the reward model and the RL loop are truly necessary, or whether the preference data can train the policy directly. They are not, and it can.\nFigure 2. The PPO loop, and the leash that keeps it honest.\n5. DPO: skipping the RL loop Direct Preference Optimization shows the RLHF objective has a closed-form optimal policy, which can be inverted to express reward in terms of the policy itself. Substituting into the Bradley–Terry likelihood yields a loss trainable by ordinary supervised gradient descent — no separate reward model, no sampling loop:\n$$ \\mathcal L_{\\text{DPO}}(\\theta)=-\\,\\mathbb E_{(x,y_w,y_l)}\\Big[\\log\\sigma\\Big(\\beta\\log\\tfrac{\\pi_\\theta(y_w\\mid x)}{\\pi_{\\text{ref}}(y_w\\mid x)}-\\beta\\log\\tfrac{\\pi_\\theta(y_l\\mid x)}{\\pi_{\\text{ref}}(y_l\\mid x)}\\Big)\\Big]. $$It directly raises the policy\u0026rsquo;s relative log-prob of preferred over rejected responses, with the same KL-to-reference anchoring baked in via $\\beta$. Simpler and more stable than PPO; widely used. What updates: $\\theta$, by a supervised-style loss on preference pairs.\nFour losses, four data sources, one mechanism. Collecting them in a single table makes the invariant impossible to miss.\n6. What updates across all stages Stage Loss / signal Parameters updated Frozen Pretraining next-token CE on web text all $\\theta$ — SFT next-token CE on demos (response-masked) all $\\theta$ (or LoRA $A,B$) $W_0$ if LoRA Reward modeling Bradley–Terry on preferences reward model $\\phi$ base LM PPO clipped policy gradient + KL policy $\\theta$ $\\phi$, $\\pi_{\\text{ref}}$ DPO preference log-sigmoid + KL policy $\\theta$ $\\pi_{\\text{ref}}$ The invariant: no new learning mechanism appears — it is gradient descent throughout, differing only in data and in where the loss signal originates (self-supervised tokens → demonstrations → human-preference comparisons). The end product is one frozen parameter set.\nInference is still read-only. None of this machinery runs when you chat with the deployed model. Your conversation does not update $\\theta$; apparent in-context \u0026ldquo;learning\u0026rdquo; is just prior tokens re-entering through the context window (Lesson 1). Genuine weight changes occur only in these deliberate developer-run stages.\nFigure 3. Four losses, four data sources, one mechanism.\nConclusion A base model knows much but does not behave as an assistant; two post-training stages fix that with the same gradient descent used for pretraining. SFT continues next-token training on curated demonstrations with the loss masked to response tokens; LoRA does this efficiently by learning a low-rank $\\frac\\alpha r BA$ correction to frozen weights. RLHF first fits a Bradley–Terry reward model $r_\\phi$ to human preference comparisons, then optimizes the policy against it with a KL-anchored PPO clipped objective (the KL leash preventing reward over-optimization); DPO collapses these into a single supervised preference loss with no reward model or RL loop. In every stage only the data and loss source change — the mechanism is gradient descent, and the result is a single frozen, aligned $\\theta$ that your conversations never alter.\nFinally, we return to inference: how the frozen model generates text, and the decoding knobs that shape it without touching a single weight.\n← Lesson 8 — Optimizers, Learning Rate \u0026amp; Batches · Course index · Next: Lesson 10 — Inference in Depth → Appendices: A — Acronyms · B — Formulas · C — Attention Variants The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/tutorials/llm-lessons/lesson-09-finetuning-sft-rlhf/","summary":"\u003ch1 id=\"lesson-9--fine-tuning-sft--rlhf\"\u003eLesson 9 — Fine-Tuning, SFT \u0026amp; RLHF\u003c/h1\u003e\n\u003cp\u003e\u003cem\u003eTechnical course on LLM mechanics — Lesson 9 of 10.\u003c/em\u003e\u003c/p\u003e\n\u003chr\u003e\n\u003ch2 id=\"recap-of-lesson-8\"\u003eRecap of Lesson 8\u003c/h2\u003e\n\u003cp\u003eAdam/AdamW maps the mini-batch gradient to a per-coordinate adaptive step via two EMAs ($\\mathbf m,\\mathbf v$), bias-corrected and divided as $\\hat{\\mathbf m}/(\\sqrt{\\hat{\\mathbf v}}+\\epsilon)$, with decoupled weight decay; its $2P$ buffers dominate training memory. A warmup→cosine-decay $\\eta$ schedule with gradient clipping, large batches, and data-scale regularization, planned via scaling laws, completes pretraining. The product is a base model: an excellent estimator of $P_\\theta(x_t\\mid x_{\\lt t})$ over web text — but not an aligned assistant. This lesson covers the post-training stages and, per the course theme, exactly what each updates. The throughline: \u003cstrong\u003eevery stage is the same gradient descent of Lessons 6–8; only the data and the loss\u0026rsquo;s source change.\u003c/strong\u003e\u003c/p\u003e","title":"Fine-Tuning, SFT and RLHF"},{"content":"Chapter 9 — How a Model Learns Throughout this book we have spoken of a model\u0026rsquo;s billions of knobs being \u0026ldquo;adjusted during training\u0026rdquo; without ever saying how. We deferred it on purpose, because the answer is the same for every architecture in the book — the storyteller, the reader, the translator, the committee, even the image-sculptor of Chapter 8. They differ in shape and in the game they play, but the engine that turns a pile of random knobs into a capable system is one and the same. This chapter is that engine, told without a single equation. It comes down to three ideas: measuring how wrong the model is, working out which way to nudge every knob, and taking a careful step in that direction — over and over, a staggering number of times.\nMeasuring wrongness You cannot improve what you cannot measure, so learning begins with a way to score how wrong the model\u0026rsquo;s output was. During training, we always know the right answer — for a storyteller it is simply the actual next word in the text, which we conveniently hid from the model; for the image-sculptor it is the noise we ourselves added. So after the model produces its guess, we compare it to the truth and boil the comparison down to a single number called the loss. A small loss means the guess was good; a large loss means it was bad.\nThe exact way this number is computed for language models is worth a moment, because it is more thoughtful than plain right-or-wrong scoring. Recall that a storyteller does not output a single word but a set of odds across all possible words. The loss looks at how much probability the model placed on the word that actually came next. If it confidently bet on the correct word, the loss is tiny. If it confidently bet on the wrong word, the loss is enormous — the penalty for being confidently wrong shoots toward the punishing extreme. This does two valuable things. It forces the model to be honest about its confidence: claiming to be ninety-nine percent sure had better be backed up by being right that often, or the model pays dearly. And it gives useful feedback even when the model is merely close, smoothly rewarding every nudge toward the right answer — whereas a flat right-or-wrong score would tell the model nothing about whether it was getting warmer.\nOne humbling fact: for language, the loss can never reach zero. Language is genuinely unpredictable — even a perfect model cannot know whether you will write \u0026ldquo;mat\u0026rdquo; or \u0026ldquo;rug\u0026rdquo; — so there is an irreducible floor of uncertainty baked into the task itself. Training closes the gap down to that floor; it can never pass it. (This same loss is often re-expressed in a friendlier form, perplexity, which you can read as \u0026ldquo;how many words is the model effectively choosing between at each step\u0026rdquo; — perplexity of one being perfect certainty and high perplexity being a model flailing among many options.)\nFigure 1. Why the score punishes confident mistakes.\nWhich way is downhill? Now the model has a single number telling it how wrong it was. The question is what to do about it: which way should each of its billions of knobs be nudged to make that number a little smaller next time?\nThe standard picture is a landscape. Imagine every possible setting of all the knobs as a location on a vast, hilly terrain, where the altitude at each location is the loss — how wrong the model is with those settings. Training is simply walking downhill. At your current spot you sense which direction slopes down most steeply, and you take a small step that way. Do this enough times and you descend from the high, error-ridden peaks where a fresh, random model begins, down into a valley of low error where the model is good. You are never guaranteed to find the single lowest valley anywhere on the whole landscape, but in these enormous models the valleys you do reach are reliably good enough. This downhill walk is called gradient descent, and it is the heart of all learning here.\nThe remaining magic is figuring out the downhill direction for all billion-plus knobs at once. Testing each knob separately — nudge it, see if the loss improves, undo it, move on — would be hopeless at this scale; you would never finish. The clever solution, called backpropagation, computes the right nudge for every knob in a single efficient backward sweep. It works backward from the final error, layer by layer down through the tower, handing each layer its fair share of the blame and reusing the work as it goes. The right mental image is an accountant tracing a company\u0026rsquo;s overall loss back to every department in one pass, rather than auditing each department from scratch. This one efficient idea is what makes training enormous models possible at all — and it is exactly why the shortcut connections and normalization from Chapter 4 matter so much, because they keep that backward-flowing blame signal from fading to nothing or blowing up as it travels down through a hundred layers.\nFigure 2. The downhill walk, and the accountant who apportions the blame.\nHow big a step, and a few refinements Knowing the downhill direction is only half the decision; you also have to choose how big a step to take. This step size, called the learning rate, is the single most consequential dial in all of training. Too big and you bound right over the valley and bounce around chaotically, never settling. Too small and training crawls for an eternity. Getting it right is much of the art.\nIn practice the step size is not held constant. Training usually opens with a gentle warm-up — tiny steps while the still-random model finds its footing, since big early steps on such chaotic terrain cause disaster — then ramps up to a brisk pace, and finally cools back down toward the end, taking small careful steps to settle precisely into the bottom of the valley. It is much like parking a car: roll up briskly, then ease gently into the spot.\nThe plain \u0026ldquo;step downhill\u0026rdquo; recipe also turns out to be too crude on its own, so the workhorse method (you may hear the name Adam) adds two refinements. The first is momentum: the process remembers its recent direction and builds up speed along consistent slopes while canceling out aimless back-and-forth jitter, like a heavy ball rolling downhill, smoothing past small bumps. The second is per-knob step sizing: each knob gets its own appropriately sized step, taking bold strides where the ground is flat and cautious ones where it is steep. These refinements are also part of why training is such a memory hog — the method has to remember extra bookkeeping for every single knob, several times over.\nOne more practical point: the model does not measure its wrongness on all of its training data before each step — that would be far too slow. It uses small random batches, getting a noisy but good-enough sense of which way is downhill from each handful of examples. The noise even helps, jostling the model out of bad spots much as shaking a pinball machine can free a stuck ball.\nFigure 3. Warm up, go briskly, then ease into the parking space.\nHow much model, how much data? Step back from the mechanics and a strategic question looms over anyone about to spend a fortune on training: given a fixed budget, should you build a bigger model with more knobs, or train a smaller one on more data? You cannot maximize both at once.\nThe encouraging discovery is that the answer is predictable. Performance improves in a smooth, forecastable way as you add computing power — so you can run small experiments, fit the trend, and reliably predict how a much larger model will perform before building it. This predictability is what lets organizations commit enormous sums with confidence. The famous correction to early practice came from a study (associated with the name Chinchilla) showing that the giant models of one era had been under-fed — too big for the amount of data they were trained on, like vast brains that hadn\u0026rsquo;t read enough. Model size and training data, it turned out, should grow roughly in step, in balanced proportion. The proof was dramatic: a model several times smaller than a rival beat it outright, simply by being trained on far more data for the same total cost.\nThere is a twist for the real world, though. What is optimal for training is not the same as what is optimal for serving. If you intend to run a model billions of times for users, a smaller model is cheaper forever — so companies often deliberately build models smaller than the balanced rule suggests and over-train them on far more data than is strictly efficient, paying extra upfront to own a permanently leaner, cheaper model. (This is the philosophy behind the smaller, heavily-trained open models many people run today.) A sobering footnote: this whole picture assumes a plentiful supply of fresh, high-quality text — and the world is genuinely beginning to run short of it, which is now a real constraint on how far the approach can be pushed.\nWhat to carry forward A model learns by a relentless three-step cycle. It scores how wrong it was with a single number, the loss, designed to punish confident mistakes and to reward every small improvement. It uses backpropagation to compute, in one efficient sweep, which way to nudge each of its billions of knobs to lower that number — a downhill walk across a vast landscape of possible settings. And it takes a carefully sized step, refined by momentum and per-knob tuning and paced by a warm-up-and-cool-down schedule. Repeat across an unfathomable amount of data, guided by predictable scaling laws that balance model size against data, and the random knobs of a newborn model settle into the configuration of a capable one.\nBut a capable predictor is not yet a helpful one. A freshly trained language model is fluent and knowledgeable and almost useless as an assistant — ask it a question and it might just continue with more questions. Turning that raw predictor into something that behaves like a helpful, well-mannered assistant is a second act of shaping, and it is the subject of Chapter 10 .\n← Chapter 8 · Diffusion Models · Contents · Chapter 10 · Raising a Helpful Assistant → The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/tutorials/inside-the-machines/chapter-09-how-a-model-learns/","summary":"\u003ch1 id=\"chapter-9--how-a-model-learns\"\u003eChapter 9 — How a Model Learns\u003c/h1\u003e\n\u003cp\u003eThroughout this book we have spoken of a model\u0026rsquo;s billions of knobs being \u0026ldquo;adjusted during training\u0026rdquo; without ever saying \u003cem\u003ehow\u003c/em\u003e. We deferred it on purpose, because the answer is the same for every architecture in the book — the storyteller, the reader, the translator, the committee, even the image-sculptor of Chapter 8. They differ in shape and in the game they play, but the engine that turns a pile of random knobs into a capable system is one and the same. This chapter is that engine, told without a single equation. It comes down to three ideas: measuring how wrong the model is, working out which way to nudge every knob, and taking a careful step in that direction — over and over, a staggering number of times.\u003c/p\u003e","title":"How a Model Learns"},{"content":"Lesson 9 — Inference: Decoding Against a Frozen Memory Technical course on encoder–decoder transformers — Lesson 9 of 10.\nRecap of Lesson 8 Training uses teacher forcing: the decoder is fed the ground-truth output shifted right by one, so all $m$ positions are scored in parallel under a causal mask, with cross-attention reading the full encoder memory. The loss is averaged cross-entropy ($\\hat{\\mathbf p}_t-\\mathbf y_t$ gradient), and that gradient is the encoder\u0026rsquo;s only learning signal, routed through cross-attention. The cost is exposure bias — the model trains on correct prefixes but must later condition on its own. We now run the frozen model: $\\theta$ is fixed, no loss, no gradient, only forward passes — and every knob is [CONFIG], every buffer [STATE].\n1. Encode once, decode many Inference splits cleanly along the two-stack boundary, and this is where the architecture pays off computationally. The encoder runs exactly once: feed the input $x$, get the memory $H_{\\text{enc}}\\in\\mathbb R^{n\\times d}$. Then the decoder runs autoregressively, one token per step, each step attending to $y_{\\lt t}$ (self) and to the frozen $H_{\\text{enc}}$ (cross):\nEncode: $H_{\\text{enc}}=\\mathrm{Enc}(x)$ — once. Start the decoder with $\\langle\\text{bos}\\rangle$. Step $t$: forward the decoder → logits $\\mathbf z_t$ → $\\hat{\\mathbf p}_t=\\mathrm{softmax}(\\mathbf z_t/T)$ at the last position. Select $y_t$ (strategies below); append it; repeat until \u0026lt;eos\u0026gt; or a length cap. Unlike training, the decoder cannot be parallelized over output positions — token $t$ genuinely depends on the sampled $y_{t-1}$. But the expensive bidirectional encoding is amortized: it is paid once and reused by all $m$ decoder steps and (for beam search) all beams.\nFigure 1. Encode once, decode many.\n2. Two KV caches, not one A decoder-only model caches the keys/values of past tokens to avoid recomputing them each step (the KV cache). An encoder–decoder has two distinct caches, because the decoder runs two kinds of attention:\nSelf-attention KV cache — keys/values of the output tokens generated so far. It grows by one each step, exactly like a decoder-only model\u0026rsquo;s cache. Size $\\sim 2\\,t\\,N_{\\text{dec}}\\,d$ at step $t$. Cross-attention KV cache — keys/values computed from $H_{\\text{enc}}$: $K=H_{\\text{enc}}W_K^c$, $V=H_{\\text{enc}}W_V^c$, per decoder layer. These depend only on the (fixed) input, so they are computed once, right after encoding, and never change during generation. Size $\\sim 2\\,n\\,N_{\\text{dec}}\\,d$, constant in $t$. $$ \\text{per-step cross-attention cost} = O(n\\,d)\\ \\text{(one query vs. } n\\text{ cached encoder keys)}, \\quad\\text{with }K,V\\text{ never recomputed.} $$This is a concrete efficiency win of the two-stack design: the source representation is encoded once and its cross-attention keys/values are cached once, so each decoding step\u0026rsquo;s conditioning on the input is cheap and the input is never re-read. The two caches are pure [STATE] — built per request, discarded after.\nflowchart TB x[\"input x\"] --\u003e enc[\"Encoder (once)\"] --\u003e Henc[\"H_enc\"] Henc --\u003e xkv[\"Cross-attn K,V cache(computed once, fixed)\"] subgraph loop[\"Decode loop (per step t)\"] step[\"forward 1 new token\"] --\u003e sa[\"self-attn: append toself KV cache (grows)\"] sa --\u003e ca[\"cross-attn: read fixedcross KV cache\"] ca --\u003e pick[\"pick y_t → append\"] end xkv -.-\u003e ca pick --\u003e step Figure 2. Two caches: one grows, one does not.\n3. Decoding strategies: why beam search, here especially The selection step (step 4) carries all behavioral variation, and the choice is task-shaped — the same spectrum as a decoder-only model, but with seq2seq\u0026rsquo;s tilt toward \u0026ldquo;there is a correct answer.\u0026rdquo;\nGreedy: $y_t=\\arg\\max_w\\hat p_w$. Fast, deterministic, but locally greedy ≠ globally optimal.\nBeam search: keep the $k$ highest-log-probability partial sequences (beams), expand each by all next tokens, keep the top $k$ overall, repeat. It approximates maximum-probability (MAP) decoding $\\arg\\max_y\\prod_t\\hat p_{y_t}$. Beam search is the default for translation and summarization (typical $k=4$–$6$) precisely because these are low-entropy tasks with a roughly-correct answer — the diversity that sampling buys is a liability when faithfulness matters. (For open-ended generation beam search produces bland text and sampling wins, but that is rarely the seq2seq setting.) Beam width $k$ is [CONFIG].\nLength normalization. Because each step multiplies in a probability $\\lt 1$, raw sequence log-prob $\\sum_t\\log\\hat p_{y_t}$ is biased toward shorter outputs — fatal for translation/summarization where length should match content. Beam search therefore divides by a length penalty:\n$$ \\text{score}(y)=\\frac{1}{\\mathrm{lp}(m)}\\sum_{t=1}^{m}\\log \\hat p_{y_t},\\qquad \\mathrm{lp}(m)=\\frac{(5+m)^\\alpha}{(5+1)^\\alpha}, $$the Google-NMT form with $\\alpha\\in[0.6,1.0]$ a [CONFIG] knob. Without it, the model truncates; with it, output length is calibrated.\nSampling (with temperature / top-k / top-p) is available and used for diverse paraphrase or data augmentation, but is the exception for core seq2seq tasks.\nFigure 3. Why beam search divides by a length penalty.\n4. Coverage: the repetition / dropping failure mode Seq2seq generation has a characteristic failure the alignment view (Lesson 4) explains: because cross-attention is recomputed freely each step with no memory of what it already attended to, the decoder can attend to the same source span repeatedly (→ repetition, \u0026ldquo;the the the\u0026rdquo;) or never attend to some span (→ under-translation / dropped content). The classic fix is a coverage mechanism: maintain a running sum of attention given to each source position so far,\n$$ \\mathbf c^{t}=\\sum_{t'=1}^{t-1}A_{t',:}\\in\\mathbb R^{n}\\quad(\\text{coverage vector}), $$then either feed $\\mathbf c^t$ into the attention computation so the model can see what it has covered, or add a coverage loss that penalizes attending to already-covered positions:\n$$ \\mathcal L_{\\text{cov}}=\\sum_{t}\\sum_{j}\\min\\big(A_{tj},\\,c^{t}_{j}\\big). $$Coverage was introduced for RNN attention (See et al.; Tu et al.) and the idea carries directly to transformer cross-attention. At transformer scale, large pretraining plus beam search with a repetition/length penalty often suffices, but coverage remains the principled account of why repetition and omission happen — uncontrolled re-attention to the source.\n5. Copy mechanisms: when the output should reuse the input Summarization and translation frequently need to reproduce exactly a token from the source — a name, a number, a rare or out-of-vocabulary word the decoder\u0026rsquo;s softmax would struggle to generate. The pointer-generator (See et al., 2017) addresses this by letting the model, at each step, either generate from the vocabulary or copy from the source by pointing at it through the cross-attention distribution. A learned soft gate $p_{\\text{gen}}\\in[0,1]$ mixes the two:\n$$ P(w)=p_{\\text{gen}}\\,P_{\\text{vocab}}(w)\\;+\\;(1-p_{\\text{gen}})\\!\\!\\sum_{j:\\,x_j=w}\\!\\! A_{tj}, \\qquad p_{\\text{gen}}=\\sigma(\\mathbf w_h^\\top\\mathbf h_t+\\mathbf w_c^\\top\\mathbf c_t+\\mathbf w_q^\\top\\mathbf q_t+b). $$The second term is literally the cross-attention weight on source positions whose token is $w$ — the alignment $A$ doubles as a copy distribution. This elegantly reuses the mechanism we already have: cross-attention both retrieves meaning (for generation) and points (for copying). Subword tokenization (BPE/SentencePiece) reduced the OOV pressure that originally motivated copying, so frontier models rely on it less, but it remains the clean answer to \u0026ldquo;the output must contain this exact source token.\u0026rdquo;\nLearned / config / state Quantity Category Frozen weights $\\theta$, $p_{\\text{gen}}$ gate (if trained) [LEARNED], never change at inference Strategy (greedy/beam/sampling), beam width $k$, length penalty $\\alpha$, $T$/top-k/top-p, max length [CONFIG] $H_{\\text{enc}}$, self-attn KV cache (grows), cross-attn KV cache (fixed), logits, coverage vector [STATE] Conclusion Inference encodes once and decodes many: the bidirectional encoder produces $H_{\\text{enc}}$ in a single pass, then the decoder generates autoregressively against it. The two-stack design yields two KV caches — a growing self-attention cache (output tokens) and a fixed cross-attention cache (computed once from $H_{\\text{enc}}$ and never recomputed), making per-step conditioning on the input cheap. Beam search with length normalization is the seq2seq default because translation and summarization are low-entropy tasks with a roughly-correct answer. The alignment matrix $A$ from cross-attention explains and powers two seq2seq-specific tools: coverage (track cumulative attention to curb repetition and dropped content) and copy / pointer-generator mechanisms (point through $A$ to reproduce exact source tokens). Every knob is configuration; every buffer is transient state; $\\theta$ is frozen — closing the train/infer loop opened in Lesson 8.\nThe mechanism is complete. Lesson 10 puts it to work: framing every task as text-to-text, fine-tuning for translation and summarization, measuring with BLEU and ROUGE, and shrinking the model by distillation into T5-small — and closes the course with the invariant that carried it.\n← Lesson 8 — Training: Teacher Forcing · Course index · Next: Lesson 10 — Fine-Tuning \u0026amp; Distillation → Appendices: A — Formulas · B — Architecture Comparison · C — Acronyms The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/tutorials/encoder-decoder-lessons/lesson-09-inference/","summary":"\u003ch1 id=\"lesson-9--inference-decoding-against-a-frozen-memory\"\u003eLesson 9 — Inference: Decoding Against a Frozen Memory\u003c/h1\u003e\n\u003cp\u003e\u003cem\u003eTechnical course on encoder–decoder transformers — Lesson 9 of 10.\u003c/em\u003e\u003c/p\u003e\n\u003chr\u003e\n\u003ch2 id=\"recap-of-lesson-8\"\u003eRecap of Lesson 8\u003c/h2\u003e\n\u003cp\u003eTraining uses \u003cstrong\u003eteacher forcing\u003c/strong\u003e: the decoder is fed the ground-truth output shifted right by one, so all $m$ positions are scored in parallel under a causal mask, with cross-attention reading the full encoder memory. The loss is averaged cross-entropy ($\\hat{\\mathbf p}_t-\\mathbf y_t$ gradient), and that gradient is the encoder\u0026rsquo;s only learning signal, routed through cross-attention. The cost is \u003cstrong\u003eexposure bias\u003c/strong\u003e — the model trains on correct prefixes but must later condition on its own. We now run the frozen model: $\\theta$ is fixed, no loss, no gradient, only forward passes — and every knob is \u003cstrong\u003e[CONFIG]\u003c/strong\u003e, every buffer \u003cstrong\u003e[STATE]\u003c/strong\u003e.\u003c/p\u003e","title":"Inference: Decoding Against a Frozen Memory"},{"content":"Lesson 10 — Conditioning, Guidance, and the Modern Landscape Recap of Lesson 9. DDIM rewrites the reverse step as \u0026ldquo;predict $\\hat x_0$ + direction term + optional $\\sigma_t z$\u0026rdquo;; with $\\sigma_t=0$ it is deterministic, lets us skip timesteps (20–50 ≈ 1000), and gives an invertible latent space. The score view shows $\\epsilon_\\theta \\approx -\\sqrt{1-\\bar\\alpha_t}\\,\\nabla_x\\log q_t(x)$, recasting generation as solving a reverse-time SDE or its probability-flow ODE (fast ODE solvers → ~10–20 steps). The $v$-parameterization keeps targets well-scaled. This final lesson adds steering and scale.\nThe model we have trained is a gifted sleepwalker. Left to itself it produces faces, landscapes, rooms — all plausible, none of them requested. It has no notion of \u0026ldquo;a red bicycle at dusk\u0026rdquo; and no reason to acquire one. This final lesson is about teaching it to take instruction: how to feed a prompt into the machine, why a naively conditioned model only half-listens, and the surprisingly sharp trick — guidance — that makes it obey. Then, because obedience at full resolution is ruinously expensive, how the entire performance is quietly moved into a smaller room.\nOverview So far we sample from the data distribution unconditionally — we cannot ask for \u0026ldquo;a photo of a red bicycle at dusk.\u0026rdquo; Real systems are conditional: given a prompt $c$ (text, class label, image) they sample from $p_\\theta(x_0\\mid c)$. Two ideas define the modern era. Guidance makes conditioning actually bite: naive conditioning under-weights $c$, and guidance amplifies it by reshaping the sampled distribution toward $p(x)p(c\\mid x)^w$. Latent diffusion runs the entire process in a compressed autoencoder space, slashing cost. Together they are the recipe behind Stable Diffusion and its descendants. We close with a rigorous orientation on the contemporary stack — DiT, distillation, and flow matching.\nConditional diffusion The framework barely changes: feed $c$ into the network, $\\epsilon_\\theta(x_t,t,c)$, and train on paired data with the same loss\n$$ L = \\mathbb{E}_{t,x_0,\\epsilon,c}\\big[\\|\\epsilon - \\epsilon_\\theta(x_t, t, c)\\|^2\\big]. $$By the same Tweedie/score argument of Lesson 9, the minimizer satisfies the conditional score identity\n$$ \\epsilon_\\theta(x_t,t,c) \\approx -\\sqrt{1-\\bar\\alpha_t}\\,\\nabla_{x_t}\\log q_t(x_t\\mid c). $$For text, $c$ is an embedding from a frozen text encoder (CLIP or T5) injected via cross-attention (detailed below). The problem: a network trained this way tends to under-use $c$, producing samples that only loosely respect the prompt — the conditional density is \u0026ldquo;too flat.\u0026rdquo; The fix is guidance.\nFigure 1. Threading a prompt through the network.\nClassifier guidance The first fix for an inattentive model came from outside the network entirely: borrow a critic who already knows the categories, and let it tug each sample into line.\nThe first approach (Dhariwal \u0026amp; Nichol, 2021) decomposes the conditional score by Bayes\u0026rsquo; rule. From $\\log q_t(x_t\\mid c)=\\log q_t(x_t)+\\log p(c\\mid x_t)-\\log p(c)$, take $\\nabla_{x_t}$ (the $\\log p(c)$ term vanishes):\n$$ \\nabla_{x_t}\\log q_t(x_t\\mid c) = \\nabla_{x_t}\\log q_t(x_t) + \\nabla_{x_t}\\log p(c\\mid x_t). $$Tilting the distribution by a power $w$ To amplify the condition, scale the classifier term by a guidance weight $w$, targeting the score of the tilted distribution $\\tilde p_w(x_t\\mid c)\\propto q_t(x_t)\\,p(c\\mid x_t)^w$:\n$$ \\nabla_{x_t}\\log \\tilde p_w(x_t\\mid c) = \\nabla_{x_t}\\log q_t(x_t) + w\\,\\nabla_{x_t}\\log p(c\\mid x_t). $$ Derivation (implied distribution). The tilt $p(c\\mid x)^w$ raises the likelihood to a power: for $w\\gt 1$ it sharpens the posterior, concentrating mass where the classifier is confident (lower entropy, higher fidelity, less diversity); $w=1$ is the true posterior; $w\\lt 1$ flattens it. The marginal over $c$ is $p_w(x)\\propto p(x)\\,\\mathbb E_{c}[\\,\\cdots]$, but per-condition the operative target is $p(x)\\,p(c\\mid x)^w$.\nConverting scores to noise via $\\epsilon=-\\sqrt{1-\\bar\\alpha_t}\\,\\nabla_{x_t}\\log(\\cdot)$ gives the guided noise estimate:\n$$ \\boxed{\\ \\hat\\epsilon = \\epsilon_\\theta(x_t,t) - w\\,\\sqrt{1-\\bar\\alpha_t}\\;\\nabla_{x_t}\\log p_\\phi(c\\mid x_t)\\ } $$Larger $w$ → stronger adherence to $c$ at the cost of diversity. The drawback: one must train and run a noise-robust classifier $p_\\phi(c\\mid x_t)$ at every noise level — extra machinery, and its gradients can be adversarially brittle.\nClassifier-free guidance (CFG) — the standard today The elegant alternative (Ho \u0026amp; Salimans, 2022) needs no separate classifier. During training, randomly drop the condition (replace $c$ with a null token $\\varnothing$) some fraction of the time (e.g. 10–20%). The same network thereby learns both the conditional $\\epsilon_\\theta(x_t,t,c)$ and the unconditional $\\epsilon_\\theta(x_t,t,\\varnothing)$.\nDeriving CFG from the score identity Apply the score identity to both. By Bayes\u0026rsquo; rule, the implicit classifier gradient is $$ \\nabla_{x_t}\\log p(c\\mid x_t)=\\nabla_{x_t}\\log q_t(x_t\\mid c)-\\nabla_{x_t}\\log q_t(x_t) \\approx -\\frac{1}{\\sqrt{1-\\bar\\alpha_t}}\\big(\\epsilon_\\theta(x_t,t,c)-\\epsilon_\\theta(x_t,t,\\varnothing)\\big). $$ Substitute this into the classifier-guidance target $\\nabla\\log q_t(x_t)+w\\,\\nabla\\log p(c\\mid x_t)$ and convert back to noise:\n$$ \\boxed{\\ \\hat\\epsilon(x_t,t,c) = \\epsilon_\\theta(x_t,t,\\varnothing) + w\\,\\big(\\epsilon_\\theta(x_t,t,c) - \\epsilon_\\theta(x_t,t,\\varnothing)\\big)\\ } $$So CFG approximately samples from $p(x)\\,p(c\\mid x)^w$ using the model\u0026rsquo;s own conditional/unconditional difference as a built-in classifier gradient — no auxiliary network.\n$w = 1$: ordinary conditional sampling. $w \\gt 1$ (typically 3–15): extrapolate past the conditional, away from the unconditional — strongly amplifying prompt adherence. This is the \u0026ldquo;guidance scale\u0026rdquo; / \u0026ldquo;CFG scale\u0026rdquo; knob in tools like Stable Diffusion. It costs two network evaluations per step (the conditional and unconditional passes can be batched).\nRemark (over-saturation and dynamic thresholding). High $w$ pushes $\\hat x_0$ outside the valid data range $[-1,1]$, producing over-saturated, blown-out images. Dynamic thresholding (Saharia et al., 2022, Imagen) rescales each predicted $\\hat x_0$ by a per-step percentile $s=\\mathrm{percentile}(|\\hat x_0|, p)$ (e.g. $p=99.5\\%$): clip to $[-s,s]$ then divide by $\\max(s,1)$. This pushes saturated pixels inward and preserves detail, enabling much larger $w$ than static clamping. The CFG weight, the schedule, and thresholding are all sampling-time choices, orthogonal to training.\nFigure 2. What obedience costs per sampling step.\nCross-attention conditioning Text conditioning is implemented by cross-attention layers interleaved with the backbone\u0026rsquo;s self-attention (Rombach et al., 2022). Let $\\phi(x_t)\\in\\mathbb R^{N\\times d}$ be flattened image/latent features ($N$ spatial tokens) and $\\tau(c)\\in\\mathbb R^{M\\times d_c}$ the text token embeddings ($M$ tokens). Then\n$$ Q=\\phi(x_t)W_Q\\;(N\\times d_k),\\quad K=\\tau(c)W_K\\;(M\\times d_k),\\quad V=\\tau(c)W_V\\;(M\\times d_v), $$ $$ \\text{Attention}=\\mathrm{softmax}\\!\\Big(\\tfrac{QK^\\top}{\\sqrt{d_k}}\\Big)V\\;\\in\\mathbb R^{N\\times d_v}. $$Queries come from image features; keys and values come from text embeddings. Each spatial location thus attends over the text tokens and pulls in the embeddings of the words relevant to it. The null token $\\varnothing$ for CFG is just an empty/learned embedding fed through the same $K,V$ projections, so CFG and cross-attention coexist seamlessly. Time $t$ is injected separately (adaptive normalization / additive embeddings), as in Lesson 6.\nLatent diffusion — making it affordable Steering solved, one obstacle still stands between us and a usable system: raw pixels are simply too expensive to denoise a thousand times over.\nDiffusion on raw $512\\times512\\times3$ pixels is enormously costly. Latent Diffusion Models (LDM) (Rombach et al., 2022 — the Stable Diffusion architecture) move the whole process into a compressed space:\nPretrain an autoencoder. An encoder $E$ maps an image to a small latent $z_0=E(x)$ (e.g. $512\\times512\\times3 \\to 64\\times64\\times4$, an $8\\times$ spatial / $\\sim48\\times$ overall compression factor), and a decoder reconstructs $x\\approx D(z_0)$. It is trained with a perceptual (LPIPS) + patch-adversarial loss for sharp reconstructions, plus a latent regularizer: either a small KL penalty toward $\\mathcal N(0,I)$ (KL-VAE, continuous latents) or a VQ codebook (discrete latents). The regularizer keeps the latent space well-scaled so diffusion is well-conditioned. Run diffusion in latent space. Apply everything from Lessons 2–9 to $z_0$ instead of $x_0$: forward noising on $z$, train $\\epsilon_\\theta(z_t,t,c)$, sample with DDIM/ODE solvers and CFG. Cross-attention and CFG live entirely inside this latent-space backbone. Decode once. Map the generated latent back to pixels with $D(z_0)$ — a single forward pass, not part of the diffusion loop. Because the latent grid is $\\sim64\\times$ smaller in element count, training and sampling are vastly cheaper — this is what put high-quality text-to-image generation on consumer GPUs.\nRemark (separation of concerns). The autoencoder is frozen during diffusion training. Reconstruction fidelity is upper-bounded by $D(E(x))$; the diffusion model only ever sees the latent manifold, so artifacts can originate either in the latent prior (diffusion) or the decoder.\nFigure 3. Move the performance into a smaller room.\nThe contemporary picture (rigorous orientation, not exhaustive) Diffusion Transformers (DiT) (Peebles \u0026amp; Xie, 2023): replace the U-Net with a transformer over latent patches. Conditioning (class, time) enters via adaptive layer norm (adaLN-zero: regress per-block scale/shift/gate from the embedding, init gates to zero). Same loss; cleaner scaling laws, now standard at frontier scale (e.g. SD3, video models). Distillation for few-step sampling. Progressive distillation (Salimans \u0026amp; Ho, 2022) trains a student to take one DDIM step matching two teacher steps, halving step count repeatedly to 1–4 steps. Consistency models (Song et al., 2023) train $f_\\theta(x_t,t)\\approx x_0$ to be self-consistent along a probability-flow ODE trajectory (any point maps to the same endpoint), enabling 1–2 step generation. Adversarial distillation (e.g. ADD/LCM-style) adds a GAN loss for sharpness. Flow matching / rectified flow (Lipman et al., 2023; Liu et al., 2022): instead of a noise schedule, regress a velocity field $v_\\theta(x_t,t)$ along prescribed probability paths (often the straight-line interpolant $x_t=(1-t)x_0+t\\,\\epsilon$). The training is simulation-free regression to the conditional velocity, and the sampler integrates an ODE $dx/dt=v_\\theta$. This is exactly the probability-flow ODE view of Lesson 9: rectified flow straightens trajectories so few Euler steps suffice. Diffusion and flow matching are two parameterizations of the same continuous-time transport. Beyond images: the identical recipe drives audio, video, 3D, molecular and protein generation — the modality only changes $\\epsilon_\\theta$ (or $v_\\theta$) and the data representation. Full course recap The arc, in one breath:\nBig picture — generation = learned denoising; fixed forward $q$, learned reverse $p_\\theta$. Forward process — Gaussian Markov chain; the closed-form jump $x_t=\\sqrt{\\bar\\alpha_t}x_0+\\sqrt{1-\\bar\\alpha_t}\\epsilon$ makes any noise level one operation away. Reverse process — small steps ⇒ Gaussian reverse; the $x_0$-conditioned posterior $q(x_{t-1}\\mid x_t,x_0)$ is tractable and, in noise form, shows predicting $\\epsilon$ suffices. Variational bound — likelihood is intractable, so optimize the ELBO; it decomposes into a constant $L_T$, reconstruction $L_0$, and the denoising KLs $L_{t-1}$ = weighted mean-matching. Simplified objective — the $\\epsilon$-parameterization cancels schedule factors, leaving $L_{\\text{simple}}=\\mathbb E\\,\\|\\epsilon-\\epsilon_\\theta(x_t,t)\\|^2$. Architecture — $\\epsilon_\\theta$ is an image-to-image U-Net (or DiT) with sinusoidal time embeddings injected per block. Training phase — a five-line loop: sample $x_0,t,\\epsilon$, jump to $x_t$, regress noise. Never runs the reverse chain. Inference (DDPM) — start at $\\mathcal N(0,I)$, iterate the stochastic reverse update down to $x_0$; ~$T$ sequential calls. Fast \u0026amp; deterministic — DDIM skips steps and can be deterministic; the score view (Tweedie) recasts it as a reverse SDE / probability-flow ODE solved fast by modern solvers. Conditioning \u0026amp; scale — cross-attention conditioning, classifier-free guidance to enforce prompts, and latent diffusion to make it all affordable — the modern text-to-image stack. The single sentence to keep: a diffusion model learns to predict the noise in a corrupted sample at every noise level; training is one-shot noising + MSE regression, and generation is iterating that predictor backwards from pure noise. Everything else — the ELBO, DDIM, the SDE, guidance, latent space — is refinement, acceleration, and control built on top of that one idea.\nExercises 1. Derive CFG. Start from the classifier-guidance target score $\\nabla\\log q_t(x)+w\\,\\nabla\\log p(c\\mid x)$ and the implicit-classifier identity $\\nabla_{x}\\log p(c\\mid x)\\approx-\\frac{1}{\\sqrt{1-\\bar\\alpha_t}}(\\epsilon_\\theta(x,t,c)-\\epsilon_\\theta(x,t,\\varnothing))$. Convert to noise via $\\epsilon=-\\sqrt{1-\\bar\\alpha_t}\\,\\nabla\\log(\\cdot)$ and obtain $\\hat\\epsilon=\\epsilon_\\varnothing+w(\\epsilon_c-\\epsilon_\\varnothing)$.\nApply $-\\sqrt{1-\\bar\\alpha_t}\\,\\nabla\\log(\\cdot)$ to the target score. The first term gives $-\\sqrt{1-\\bar\\alpha_t}\\,\\nabla\\log q_t(x)=\\epsilon_\\theta(x,t,\\varnothing)=\\epsilon_\\varnothing$. The second gives $-\\sqrt{1-\\bar\\alpha_t}\\cdot w\\cdot\\big(-\\frac{1}{\\sqrt{1-\\bar\\alpha_t}}\\big)(\\epsilon_c-\\epsilon_\\varnothing)=w(\\epsilon_c-\\epsilon_\\varnothing)$. Summing, $\\hat\\epsilon=\\epsilon_\\varnothing+w(\\epsilon_c-\\epsilon_\\varnothing)$.\n2. What CFG samples, and the trade-off. Which (unnormalized) distribution does CFG approximately sample from, and what does increasing $w$ above $1$ do to fidelity, diversity, and pixel saturation?\nIt approximately targets the tilted distribution $p(x)\\,p(c\\mid x)^w$. For $w\\gt 1$ the posterior is sharpened: stronger prompt adherence (higher fidelity) at the cost of lower diversity, and $\\hat x_0$ tends to push outside $[-1,1]$, causing over-saturated/blown-out images — which is why high $w$ is paired with dynamic thresholding. $w=1$ is the ordinary conditional; $w\\lt 1$ flattens it.\n3. Cross-attention shapes. With $Q\\in\\mathbb R^{N\\times d_k}$, $K\\in\\mathbb R^{M\\times d_k}$, $V\\in\\mathbb R^{M\\times d_v}$, verify the attention output is $N\\times d_v$, and state which of $N,M$ counts image tokens vs. text tokens.\n$QK^\\top$ is $N\\times M$; after softmax (row-wise over the $M$ keys) it stays $N\\times M$; multiplying by $V$ ($M\\times d_v$) gives $N\\times d_v$. Here $N$ is the number of image/latent spatial tokens (queries come from image features) and $M$ is the number of text tokens (keys and values come from the text embedding). Each spatial location thus attends over the words.\n4. Why latent diffusion is cheap. A $512\\times512\\times3$ image is encoded to a $64\\times64\\times4$ latent. Estimate the reduction in spatial element count and explain why this disproportionately helps the attention layers.\nSpatial positions drop from $512^2=262{,}144$ to $64^2=4096$ — a $64\\times$ reduction ($8\\times$ per side; channels $3\\to4$ are negligible). Per-step convolutional cost scales with the number of positions, but self-attention scales as $O((HW)^2)$, so a $64\\times$ drop in $HW$ cuts attention cost by up to $\\sim\\!64^2\\approx4000\\times$ at a given resolution. Diffusion runs entirely on the small grid and the decoder is applied once, outside the loop — which is what put high-res text-to-image on consumer GPUs.\nConclusion Conditioning threads a prompt $c$ through the network via cross-attention (image queries, text keys/values), trained with the same simple loss. Classifier-free guidance — train with random condition dropout, then extrapolate $\\hat\\epsilon=\\epsilon_\\varnothing+w(\\epsilon_c-\\epsilon_\\varnothing)$ — approximately samples from the tilted distribution $p(x)\\,p(c\\mid x)^w$ and is the standard lever for prompt adherence, paired with dynamic thresholding at high $w$. Latent diffusion runs the entire machinery in a compressed, regularized autoencoder space, which made high-resolution text-to-image practical. DiT, distillation, and flow matching extend the same continuous-time transport to larger scales and fewer steps. With the recap above, you now hold the complete chain of reasoning from \u0026ldquo;pour sand on a photo\u0026rdquo; to a modern generative system — both the why (variational, score-based) and the how (the training and sampling algorithms). That concludes the course.\nThe sleepwalker takes requests now, and does so affordably. From \u0026ldquo;pour sand on a photograph\u0026rdquo; to a system that paints a red bicycle at dusk on command, the chain of reasoning is complete — and it was, every link of it, the same single idea seen from a new angle each time. That is the end of the course; the rest is yours to build.\nReferences P. Dhariwal, A. Nichol. Diffusion Models Beat GANs on Image Synthesis (classifier guidance). NeurIPS 2021. J. Ho, T. Salimans. Classifier-Free Diffusion Guidance. NeurIPS Workshop 2021 / 2022. R. Rombach, A. Blattmann, D. Lorenz, P. Esser, B. Ommer. High-Resolution Image Synthesis with Latent Diffusion Models (LDM / Stable Diffusion). CVPR 2022. C. Saharia et al. Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding (Imagen, dynamic thresholding). NeurIPS 2022. W. Peebles, S. Xie. Scalable Diffusion Models with Transformers (DiT). ICCV 2023. Y. Song, P. Dhariwal, M. Chen, I. Sutskever. Consistency Models. ICML 2023. T. Salimans, J. Ho. Progressive Distillation for Fast Sampling of Diffusion Models. ICLR 2022. Y. Lipman, R. T. Q. Chen, H. Ben-Hamu, M. Nickel, M. Le. Flow Matching for Generative Modeling. ICLR 2023. X. Liu, C. Gong, Q. Liu. Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow. 2022. ← Previous: Lesson 9 — DDIM \u0026amp; the Score View · Course Index · Appendix A: Acronyms · Appendix B: Formulas End of course.\nThe figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/tutorials/diffusion-lessons/lesson-10-conditioning-and-guidance/","summary":"\u003ch1 id=\"lesson-10--conditioning-guidance-and-the-modern-landscape\"\u003eLesson 10 — Conditioning, Guidance, and the Modern Landscape\u003c/h1\u003e\n\u003cblockquote\u003e\n\u003cp\u003e\u003cstrong\u003eRecap of Lesson 9.\u003c/strong\u003e DDIM rewrites the reverse step as \u0026ldquo;predict $\\hat x_0$ + direction term + optional $\\sigma_t z$\u0026rdquo;; with $\\sigma_t=0$ it is deterministic, lets us skip timesteps (20–50 ≈ 1000), and gives an invertible latent space. The \u003cstrong\u003escore view\u003c/strong\u003e shows $\\epsilon_\\theta \\approx -\\sqrt{1-\\bar\\alpha_t}\\,\\nabla_x\\log q_t(x)$, recasting generation as solving a reverse-time \u003cstrong\u003eSDE\u003c/strong\u003e or its \u003cstrong\u003eprobability-flow ODE\u003c/strong\u003e (fast ODE solvers → ~10–20 steps). The $v$-parameterization keeps targets well-scaled. This final lesson adds steering and scale.\u003c/p\u003e","title":"Conditioning and Guidance"},{"content":"Lesson 10 — Fine-Tuning, Task Framing \u0026amp; Distillation Technical course on encoder–decoder transformers — Lesson 10 of 10.\nRecap of Lesson 9 Inference encodes once, decodes many: the encoder builds $H_{\\text{enc}}$ in one pass; the decoder generates autoregressively with two KV caches — a growing self-attention cache and a fixed cross-attention cache. Beam search with length normalization is the seq2seq default; the alignment matrix $A$ powers coverage (anti-repetition) and copy/pointer-generator mechanisms. $\\theta$ is frozen, every knob [CONFIG], every buffer [STATE]. We now take the pretrained model and make it useful: task framing, fine-tuning, evaluation, and compression.\n1. Text-to-text: one interface for every task T5\u0026rsquo;s organizing idea is that every NLP task can be cast as \u0026ldquo;text in → text out\u0026rdquo;, which is exactly the shape an encoder–decoder already has. Rather than attach task-specific heads (a classification head, a span head, a regression head — the encoder-only/BERT approach), T5 encodes the task itself into the input as a natural-language prefix, and reads the answer off the decoder as text:\n$$ \\begin{aligned} \\texttt{\"translate English to German: That is good.\"} \u0026\\;\\to\\; \\texttt{\"Das ist gut.\"}\\\\ \\texttt{\"summarize: \\textless{}article…\\textgreater{}\"} \u0026\\;\\to\\; \\texttt{\"\\textless{}summary…\\textgreater{}\"}\\\\ \\texttt{\"cola sentence: The course is jumps well.\"} \u0026\\;\\to\\; \\texttt{\"unacceptable\"}\\\\ \\texttt{\"stsb sentence1: … sentence2: …\"} \u0026\\;\\to\\; \\texttt{\"3.8\"}\\ (\\text{a number, as text}). \\end{aligned} $$The payoff is uniformity: the same model, same loss, same decoding handle translation, summarization, classification, and regression — only the data and the prefix change. Classification becomes \u0026ldquo;generate the class name\u0026rdquo;; regression becomes \u0026ldquo;generate the number as a string.\u0026rdquo; This is a [CONFIG]/data convention, not an architecture change — the cross-attention core of Lessons 4–5 is untouched. (BART, lacking this framing, is fine-tuned the more conventional way: feed source to the encoder, target to the decoder, with task-specific data.)\nFigure 1. One interface: text in, text out.\n2. Fine-tuning: same machinery, real pairs Fine-tuning a pretrained encoder–decoder is mechanically identical to pretraining (Lessons 7–8) with two changes: the data is now labeled (input, output) pairs for the target task, and the corruption step is gone — the encoder reads the real source, the decoder is teacher-forced on the real target.\n$$ \\mathcal L_{\\text{ft}}(\\theta)=-\\frac{1}{m}\\sum_{t=1}^{m}\\log P_\\theta\\big(y_t\\mid y_{\\lt t},\\,x\\big), $$with $(x,y)$ a translation pair (WMT), an (article, summary) pair (CNN/DailyMail, XSum), or a (task-prefixed input, answer) pair. All parameters are typically updated, though parameter-efficient methods (LoRA, adapters, prompt-tuning) apply unchanged — they are agnostic to whether the backbone is decoder-only or encoder–decoder. The learning rate is much smaller than pretraining, and far fewer steps are needed because the representations are already good; cross-attention now specializes from \u0026ldquo;reconstruct corrupted text\u0026rdquo; to \u0026ldquo;align source to target for this task.\u0026rdquo;\n3. Measuring generation: BLEU and ROUGE Cross-entropy/perplexity measure how well the model predicts the reference token-by-token, but seq2seq outputs are judged by whether the generated text matches a reference, allowing for many valid phrasings. Two n-gram overlap metrics dominate, and the choice tracks the task\u0026rsquo;s bias toward precision or recall.\nBLEU (translation) is precision-oriented: of the n-grams the model produced, how many appear in the reference? It multiplies modified n-gram precisions (typically $n=1\\dots4$) and applies a brevity penalty to stop the model gaming precision by emitting very short outputs:\n$$ \\text{BLEU}=\\underbrace{\\min\\!\\Big(1,\\,e^{1-r/c}\\Big)}_{\\text{brevity penalty}}\\cdot\\exp\\!\\Big(\\sum_{n=1}^{4} w_n\\log p_n\\Big), $$with $c$ the candidate length, $r$ the reference length, $p_n$ the modified n-gram precision. The brevity penalty is the metric-side analogue of the length normalization of Lesson 9 — both exist because probability-maximizing decoders are biased toward shortness.\nROUGE (summarization) is recall-oriented: of the reference\u0026rsquo;s n-grams (ROUGE-N) or its longest common subsequences (ROUGE-L), how many did the summary recover? Summarization cares that salient content is covered, so recall is the natural lens. Both metrics are crude (they reward surface overlap, not meaning) and are increasingly supplemented by learned metrics (BERTScore, COMET), but BLEU and ROUGE remain the reporting standard for translation and summarization respectively.\nFigure 2. The brevity penalty, and why the metric needs one.\n4. Distillation: shrinking the model A pretrained encoder–decoder is large and its autoregressive decoder is the latency bottleneck at inference. Knowledge distillation trains a small student (e.g. T5-small, DistilBART) to imitate a large teacher, transferring capability into far fewer parameters. The student minimizes a blend of the true cross-entropy and a term matching the teacher\u0026rsquo;s softened distribution:\n$$ \\mathcal L_{\\text{KD}}=(1-\\lambda)\\,\\underbrace{\\mathrm{CE}(y,\\hat{\\mathbf p}^{\\,s})}_{\\text{hard labels}} +\\lambda\\,T^2\\,\\underbrace{D_{\\mathrm{KL}}\\!\\big(\\hat{\\mathbf p}^{\\,t}_{T}\\,\\|\\,\\hat{\\mathbf p}^{\\,s}_{T}\\big)}_{\\text{soft teacher targets}}, $$where $\\hat{\\mathbf p}_T=\\mathrm{softmax}(\\mathbf z/T)$ are temperature-softened probabilities (the $T^2$ rescales the soft-target gradient to match the hard-label scale), and $\\lambda$ is a [CONFIG] mixing weight. The teacher\u0026rsquo;s full distribution carries more information than the one-hot label — \u0026ldquo;the next token is Das with 0.7, Es with 0.2, …\u0026rdquo; tells the student about near-misses — so the student learns faster and better than from hard labels alone.\nTwo seq2seq-specific wrinkles. Sequence-level distillation (Kim \u0026amp; Rush, 2016) is the dominant practice: rather than match per-token distributions, run the teacher with beam search to produce high-quality target sequences, then train the student with ordinary teacher-forced CE on those teacher-generated targets — simpler and very effective for translation. And one can distill the stacks asymmetrically: DistilBART, for instance, keeps the full encoder but copies only a subset of decoder layers, because the autoregressive decoder dominates inference latency (it runs $m$ times; the encoder runs once, Lesson 9). The architecture\u0026rsquo;s compute asymmetry directly shapes how it is compressed.\nLearned / config / state Quantity Category Fine-tuned/distilled student weights, $p_{\\text{gen}}$ gate [LEARNED] Task prefixes, fine-tuning LR/steps, distillation $\\lambda$ and $T$, beam width for seq-level KD [CONFIG] Teacher soft targets / generated sequences, logits, KV caches [STATE] Figure 3. Sequence-level distillation, and the asymmetry it exploits.\nConclusion A pretrained encoder–decoder becomes useful through text-to-text framing (T5): encode the task as an input prefix and read the answer as decoder text, so one model, loss, and decoder serve translation, summarization, classification, and regression alike. Fine-tuning is the training loop of Lesson 8 on real (input, output) pairs with no corruption; BLEU (precision, brevity-penalized) and ROUGE (recall, n-gram/LCS) score the generated text against references. Distillation — especially sequence-level KD, and asymmetric student designs that keep the encoder but thin the latency-dominant decoder — compresses the model into a T5-small or DistilBART. Throughout, the cross-attention core and the loss are untouched: only data, framing, and size change.\nThe whole course in one paragraph A seq2seq task is the conditional map $P_\\theta(y\\mid x)=\\prod_t P_\\theta(y_t\\mid y_{\\lt t},x)$ (L1), met by two stacks: a bidirectional encoder that reads $x$ once into the memory $H_{\\text{enc}}$ (L2) and a causal decoder that generates $y$ (L3) while reaching into that memory through cross-attention — queries from the decoder, keys/values from the encoder, a rectangular $m\\times n$ alignment, the one new mechanism (L4). Assembled into three-sublayer decoder blocks (L5), the design is one point on a masking spectrum with the prefix-LM and decoder-only nearby (L6). It is pretrained by denoising — T5 span corruption, BART\u0026rsquo;s five noises (L7) — via teacher forcing, with the loss gradient routed through cross-attention to train both stacks jointly (L8); it is run by encoding once and decoding many against two KV caches, with beam search, coverage, and copy mechanisms (L9); and it is specialized and shrunk by fine-tuning and distillation (L10). One invariant carried all ten lessons: the encoder compresses context, the decoder generates conditioned on it, and cross-attention is the only new mechanism — plus the two inherited ones, training writes / inference reads and parameters learned, configuration chosen, state transient.\n← Lesson 9 — Inference · Course index · Next: (none — final lesson)\nAppendices: A — Formulas · B — Architecture Comparison · C — Acronyms The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/tutorials/encoder-decoder-lessons/lesson-10-finetuning-and-distillation/","summary":"\u003ch1 id=\"lesson-10--fine-tuning-task-framing--distillation\"\u003eLesson 10 — Fine-Tuning, Task Framing \u0026amp; Distillation\u003c/h1\u003e\n\u003cp\u003e\u003cem\u003eTechnical course on encoder–decoder transformers — Lesson 10 of 10.\u003c/em\u003e\u003c/p\u003e\n\u003chr\u003e\n\u003ch2 id=\"recap-of-lesson-9\"\u003eRecap of Lesson 9\u003c/h2\u003e\n\u003cp\u003eInference \u003cstrong\u003eencodes once, decodes many\u003c/strong\u003e: the encoder builds $H_{\\text{enc}}$ in one pass; the decoder generates autoregressively with \u003cstrong\u003etwo KV caches\u003c/strong\u003e — a growing self-attention cache and a \u003cem\u003efixed\u003c/em\u003e cross-attention cache. \u003cstrong\u003eBeam search with length normalization\u003c/strong\u003e is the seq2seq default; the alignment matrix $A$ powers \u003cstrong\u003ecoverage\u003c/strong\u003e (anti-repetition) and \u003cstrong\u003ecopy/pointer-generator\u003c/strong\u003e mechanisms. $\\theta$ is frozen, every knob \u003cstrong\u003e[CONFIG]\u003c/strong\u003e, every buffer \u003cstrong\u003e[STATE]\u003c/strong\u003e. We now take the pretrained model and make it useful: task framing, fine-tuning, evaluation, and compression.\u003c/p\u003e","title":"Fine-Tuning, Task Framing \u0026 Distillation"},{"content":"Lesson 10 — Inference in Depth Technical course on LLM mechanics — Lesson 10 of 10.\nRecap of Lesson 9 Post-training aligns the base model via the same gradient descent: SFT (response-masked next-token CE on demonstrations; LoRA for efficiency), then RLHF — a Bradley–Terry reward model $r_\\phi$ followed by KL-anchored PPO, or the single-loss DPO — each updating only the policy or reward model, leaving the mechanism unchanged. The output is one frozen, aligned $\\theta$. We now return to inference, the regime where $\\theta$ is fixed and only the forward pass runs, and specify every decoding knob and the systems machinery that makes generation fast. Per the course theme: nothing here updates a parameter; every knob is [CONFIG] and every buffer is [STATE].\nLesson 1 opened the box by promising two phases — training writes, inference reads — and the eight lessons since have been spent almost entirely on the writing. The model is now finished and frozen. This final lesson watches it speak: the same forward pass we have built up component by component, run in a loop, with a handful of dials that decide how a fixed distribution gets turned into text, and the systems tricks that make that loop fast enough to serve. Nothing learns here; we are reading the function we spent nine lessons constructing.\n1. Autoregressive generation Generation samples from the factorized distribution $P_\\theta(y_{1:m}\\mid x)=\\prod_t P_\\theta(y_t\\mid x,y_{\\lt t})$ one token at a time:\nForward the context → logits $\\mathbf z$ → $\\hat{\\mathbf p}=\\mathrm{softmax}(\\mathbf z/T)$ at the last position. Select $y_t$ from $\\hat{\\mathbf p}$ (strategies below). Append $y_t$; repeat until an end-of-sequence token or length cap. Every iteration is a pure forward pass — no loss, no gradient (Lesson 1). All behavioral variation lives in step 2 and in the temperature/truncation applied before it.\nSince the forward pass is fixed, the entire art of generation is concentrated in that one selection step. We start with the strategies for which token to take from the distribution.\nFigure 1. The generation loop: a forward pass, a choice, and repeat.\n2. Decoding strategies Greedy: $y_t=\\arg\\max_w\\hat p_w$. Deterministic, but locally greedy ≠ globally optimal — $\\arg\\max$ per step does not maximize sequence probability $\\prod_t\\hat p_{y_t}$, and it tends to degenerate loops.\nBeam search: maintain the $k$ highest-log-probability prefixes, expanding and pruning to $k$ each step; approximate MAP decoding. Strong for low-entropy tasks with a \u0026ldquo;correct\u0026rdquo; answer (translation, closed QA); for open-ended generation it produces bland, repetitive text, because the highest-probability sequence is often generic. Beam width $k$ is [CONFIG].\nSampling: draw $y_t\\sim\\hat{\\mathbf p}$. Restores diversity and matches the model\u0026rsquo;s distribution, but the heavy tail — thousands of low-probability tokens whose mass is collectively non-trivial — occasionally yields incoherent picks. The next three knobs reshape or truncate that distribution before that draw, trading off the tail\u0026rsquo;s diversity against its incoherence.\n3. Temperature, top-k, top-p Temperature rescales logits before softmax:\n$$ \\hat p_w(T)=\\frac{e^{z_w/T}}{\\sum_m e^{z_m/T}}. $$$T\\to0$ → argmax (sharp/deterministic); $T=1$ → trained distribution; $T\\gt 1$ → flatter (more diverse, eventually incoherent). Effect on the earlier $\\mathbf z=[2,1,0.1]$ example: top-token mass is $0.659$ at $T{=}1$, $\\approx0.84$ at $T{=}0.5$, $\\approx0.50$ at $T{=}2$. Same weights, reweighted reading.\nTop-k: restrict to the $k$ highest-probability tokens, renormalize, sample. Simple but rigid — a fixed $k$ is too permissive when the model is confident and too restrictive when it is genuinely uncertain.\nTop-p (nucleus): restrict to the smallest set $S$ with cumulative mass $\\ge p$, renormalize, sample:\n$$ S=\\arg\\min\\big\\{|S|:\\textstyle\\sum_{w\\in S}\\hat p_w\\ge p\\big\\}. $$The candidate set size adapts to the model\u0026rsquo;s confidence — narrow when one token dominates, wide when the distribution is flat — which is why nucleus sampling is the common default. Often composed: temperature → top-p/top-k → sample. Related: repetition/frequency penalties subtract from logits of recently used tokens to curb loops; min-p thresholds relative to the peak. All [CONFIG]; none touch $\\theta$.\nThat settles what the loop emits. The remaining sections are about making the loop affordable — because a naive implementation of \u0026ldquo;forward the context each step\u0026rdquo; is quadratically wasteful, and at the scale of real serving that waste is the difference between feasible and not.\nFigure 2. Three ways to reshape the distribution before the draw.\n4. The KV cache — why generation is tractable Naive generation reforwards the entire growing prefix each step, recomputing all keys/values — $O(n^2)$ work per token, $O(n^3)$ for a sequence. But by causal masking, the keys and values of past tokens are invariant as new tokens arrive. The KV cache stores, per layer per head, the $K,V$ of all processed tokens; each step computes $Q,K,V$ only for the one new token, appends its $K,V$, and attends against the cache:\n$$ \\text{per-step cost: } O(n\\,d)\\ \\text{(one query vs. } n \\text{ cached keys)}\\quad\\text{vs. naive } O(n^2 d). $$This splits generation into two regimes: prefill (process the whole prompt in parallel, one big forward pass, compute-bound) and decode (one token at a time using the cache, memory-bandwidth-bound). The cache is pure [STATE] — built per request, discarded after.\nKV-cache memory is the binding constraint for long context:\n$$ \\text{bytes}=2\\times n\\times N_{\\text{layers}}\\times d\\times (\\text{bytes/elem}), $$(the 2 is K and V) — linear in sequence length and depth, and at long $n$ it dominates total inference memory. Mitigations reduce the $d$ factor by sharing keys/values across query heads: Multi-Query Attention (MQA) uses one K/V head for all query heads; Grouped-Query Attention (GQA) shares K/V across small groups — both shrink the cache (and decode bandwidth) by the sharing factor with minimal quality loss, which is why GQA is now standard in large models.\nThe KV cache attacks the cost of attending over a long history. The remaining latency wins attack the other bottleneck of decode — that each token requires loading the full weight matrix from memory — by doing more useful work per such load.\nFigure 3. What the KV cache removes, and what it costs instead.\n5. Latency optimizations (brief) Speculative decoding: a small \u0026ldquo;draft\u0026rdquo; model proposes several tokens; the large model verifies them in one parallel forward pass and accepts the longest correct prefix. Output distribution is provably unchanged; wall-clock speedups come from verifying many tokens per large-model call. Quantization: store/compute weights (and sometimes KV cache) in 8-, 4-, or fewer bits. Cuts memory and bandwidth — the decode bottleneck — with small, manageable accuracy loss. This changes the representation of $\\theta$, not its learned values. Batching/continuous batching: serve many requests together to amortize the memory-bandwidth cost of loading weights, raising throughput. Pulling these knobs and buffers together one last time sorts every quantity at inference into the three categories that have organized the entire course.\n6. The complete inference taxonomy At inference Category Parameters $\\theta$ [LEARNED], frozen — never change Temperature, top-k, top-p, penalties [CONFIG] (shape sampling) Strategy (greedy/beam/sampling), beam width $k$ [CONFIG] Max tokens, stop sequences [CONFIG] Logits, probabilities, KV cache [STATE] The decisive illustration of the course\u0026rsquo;s theme: two deployments of identical weights can be terse and deterministic ($T\\!\\to\\!0$, greedy) or wild and creative (high $T$, wide top-p) — same $\\theta$, different config. Parameters are the model; configuration is how you run it.\nConclusion Inference is a loop of forward passes that samples the autoregressive factorization. Greedy and beam search approximate maximum-probability decoding (good for closed tasks, bland for open ones); sampling restores diversity. Temperature sharpens or flattens the distribution; top-k and adaptive top-p truncate the tail. The KV cache exploits the invariance of past keys/values to make per-step cost $O(nd)$ instead of $O(n^2d)$, splitting work into compute-bound prefill and bandwidth-bound decode; its $2nN_{\\text{layers}}d$ memory drives long-context engineering, mitigated by GQA/MQA, speculative decoding, and quantization. Every knob is configuration and every buffer is transient state — at inference $\\theta$ is completely frozen, closing the loop opened in Lesson 1.\nAnd with that the arc is complete. We began with a function $f(x;\\theta)$ and a promise of two phases; we opened the box bottom-up, defined and measured \u0026ldquo;good,\u0026rdquo; built the engine that wrote $\\theta$, reshaped the base model into an assistant, and have now watched the frozen result generate. Everything you have seen reduces to one object run in two regimes — and to the two distinctions that told you, at every step, which part of that object you were looking at.\nThe whole course in one paragraph Text becomes token IDs (fixed BPE) then embeddings $E$ (L2); self-attention $\\mathrm{softmax}(QK^\\top/\\sqrt{d_k}+M)V$ mixes context (L3) inside residual + normalized blocks whose MLPs hold most parameters and supply nonlinearity (L4); the unembedding + softmax emit next-token probabilities (L5). Training minimizes cross-entropy $=H(p)+D_{\\mathrm{KL}}$ (L6); reverse-mode backprop computes $\\nabla_\\theta\\mathcal L$ via VJPs seeded by $\\hat{\\mathbf p}-\\mathbf y$ (L7), and AdamW with a warmup-cosine schedule descends it, planned by scaling laws (L8). SFT + RLHF/DPO then realign the base model by the same gradient descent (L9), and inference generates from the frozen result via sampling knobs and a KV cache (L10). Two distinctions carried us throughout and are worth keeping: training writes, inference reads, and parameters are learned, configuration is chosen, state is transient.\n← Lesson 9 — Fine-Tuning, SFT \u0026amp; RLHF · Course index · Next: (none — final lesson)\nAppendices: A — Acronyms · B — Formulas · C — Attention Variants The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/tutorials/llm-lessons/lesson-10-inference-in-depth/","summary":"\u003ch1 id=\"lesson-10--inference-in-depth\"\u003eLesson 10 — Inference in Depth\u003c/h1\u003e\n\u003cp\u003e\u003cem\u003eTechnical course on LLM mechanics — Lesson 10 of 10.\u003c/em\u003e\u003c/p\u003e\n\u003chr\u003e\n\u003ch2 id=\"recap-of-lesson-9\"\u003eRecap of Lesson 9\u003c/h2\u003e\n\u003cp\u003ePost-training aligns the base model via the same gradient descent: \u003cstrong\u003eSFT\u003c/strong\u003e (response-masked next-token CE on demonstrations; LoRA for efficiency), then \u003cstrong\u003eRLHF\u003c/strong\u003e — a Bradley–Terry \u003cstrong\u003ereward model\u003c/strong\u003e $r_\\phi$ followed by KL-anchored \u003cstrong\u003ePPO\u003c/strong\u003e, or the single-loss \u003cstrong\u003eDPO\u003c/strong\u003e — each updating only the policy or reward model, leaving the mechanism unchanged. The output is one frozen, aligned $\\theta$. We now return to inference, the regime where $\\theta$ is fixed and only the forward pass runs, and specify every decoding knob and the systems machinery that makes generation fast. Per the course theme: \u003cstrong\u003enothing here updates a parameter; every knob is [CONFIG] and every buffer is [STATE].\u003c/strong\u003e\u003c/p\u003e","title":"Inference in Depth"},{"content":"Lesson 10 — Inference in Depth Technical course on MoE LLM mechanics — Lesson 10 of 10.\nRecap of Lesson 9 Adapting an MoE uses the same SFT/RLHF/DPO machinery as a dense model, but sparsity makes it overfit (capacity is the large $P_{\\text{tot}}$ on small data, with uneven expert gradient coverage), countered by expert dropout, selective/LoRA fine-tuning, and the uniquely-MoE option of freezing the router. The output is one frozen, aligned $\\theta$. We now serve it. As in the companion course, nothing here updates a parameter; every knob is [CONFIG] and every buffer is [STATE] — but the Lesson 1 memory-vs-compute asymmetry now becomes the central, practical fact of inference.\nThe dense course\u0026rsquo;s final lesson watched a finished model speak: a loop of forward passes, a few sampling knobs, a KV cache to make it fast. All of that carries over verbatim — the sampling knobs and the KV cache are properties of attention and the output head, which MoE leaves untouched. This lesson is therefore short on what stays the same and focused on the one thing that changes: an MoE must hold all its experts but compute only a few, and that gap reshapes the economics and engineering of serving.\n1. What is identical to dense inference Generation is the same autoregressive loop: forward the context → logits $\\mathbf z$ → $\\hat{\\mathbf p}=\\mathrm{softmax}(\\mathbf z/T)$ → select $y_t$ → append → repeat. Every decoding strategy and knob from the dense course applies unchanged, because none of them touch the FFN/MoE sub-layer:\nDecoding: greedy, beam, sampling. Distribution shaping: temperature $T$, top-$k$, top-$p$ (nucleus), repetition penalties. The KV cache: generation still caches past keys/values per layer per head, splitting into compute-bound prefill and bandwidth-bound decode; its memory is still $\\approx 2\\,n\\,N_{\\text{layers}}\\,d\\times(\\text{bytes/elem})$, mitigated by GQA/MQA. The KV cache is a function of attention, which MoE does not change — so MoE has exactly the same KV cache as a dense model of the same attention shape. It is worth stating plainly because it is a common confusion: sparsity is in the FFN, not in attention. The expert mechanism adds nothing to the KV cache and nothing to the sampling knobs. Everything MoE-specific at inference is about loading and running the experts.\n2. The memory-vs-compute asymmetry, at serving time This is the headline. Recall Lesson 1: compute follows $P_{\\text{act}}$, memory follows $P_{\\text{tot}}$. At inference that becomes:\n$$ \\text{per-token FLOPs}\\approx 2\\,P_{\\text{act}}\\qquad\\text{but}\\qquad \\text{resident memory}\\approx (\\text{bytes/param})\\times P_{\\text{tot}}. $$So an MoE serves at the speed of a small model but demands the memory of a large one. Mixtral-8×7B computes like a 13B model ($P_{\\text{act}}$) but must keep all $\\approx47\\mathrm B$ parameters ($P_{\\text{tot}}$) in memory; DeepSeek-V3 computes like a 37B model but holds 671B. Two consequences define MoE serving:\nYou cannot drop the unused experts. Routing is data-dependent and decided per token at runtime — over a stream of tokens, every expert will eventually be needed. So the full $P_{\\text{tot}}$ must be resident (or fetchable), even though any single token touches only $k/E$ of it. Decode is even more memory-bandwidth-bound than usual. Autoregressive decode is already dominated by the cost of loading weights from memory rather than the multiply-adds (the dense course\u0026rsquo;s Lesson 10). An MoE makes this sharper: it has many experts to potentially load, and at small batch sizes a token may activate experts whose weights must be fetched for just that token — poor compute-per-byte. MoE\u0026rsquo;s FLOP savings are real, but they convert a compute problem into a memory-traffic problem. Contrast with the dense model. A dense model\u0026rsquo;s inference memory and compute are the same number $P$ — fit it and it runs. An MoE decouples them: it is cheap to compute and expensive to hold, so the binding constraint at serving shifts from FLOPs to memory capacity and bandwidth. A dense 13B model and Mixtral cost similar FLOPs per token, but Mixtral needs ~3.5× the memory to be served. Choosing an MoE is choosing to spend memory to get quality at low compute — the inference-side restatement of the Lesson 8 scaling trade-off.\nFigure 1. A small model\u0026rsquo;s speed, a large model\u0026rsquo;s memory.\n3. Batched routing and load imbalance at inference Serving batches many requests together (continuous batching, from the dense course) to amortize weight loading. For an MoE this interacts with routing: across a batch, tokens scatter to experts unevenly, so the same load-imbalance and capacity issues from training (Lessons 5–6) reappear at inference. A popular expert may be a throughput bottleneck for the batch; a fixed inference capacity may drop tokens (usually avoided at inference by using a larger capacity factor or dropless kernels, since quality matters more than strict shape there). Expert-parallel serving — sharding experts across devices, with the same all-to-all dispatch/combine of Lesson 7 — is used for the largest MoEs, making their inference, like their training, partly communication-bound. The router still runs in higher precision for the same stability reason as in training (Lesson 8).\nFigure 2. Training\u0026rsquo;s balance problems, met again at serving time.\n4. Serving optimizations specific to MoE On top of the dense toolkit (speculative decoding, quantization, batching), MoE serving adds techniques aimed squarely at the memory problem:\nExpert offloading. Keep hot/shared experts in fast accelerator memory and cold experts in CPU RAM (or disk), fetching them on demand. This trades latency for the ability to serve a model whose $P_{\\text{tot}}$ exceeds accelerator memory — viable because any one token needs only $k$ experts, and prefetching can hide some of the transfer. Quantization of experts. Since experts are the overwhelming majority of $P_{\\text{tot}}$ (Lesson 2), quantizing them to 4-bit (or lower) is the highest-leverage way to cut the resident-memory cost — and MoEs are often more tolerant of expert quantization because each expert handles a narrower slice of inputs. As in the dense course, this changes the representation of $\\theta$, not its learned values. Expert-parallel inference. Spread experts over devices and route with all-to-all, as in training — the standard way to serve frontier MoEs whose $P_{\\text{tot}}$ needs many accelerators. Figure 3. Three techniques aimed squarely at the memory problem.\n5. The complete inference taxonomy At inference Category Parameters $\\theta$ (all experts + router), frozen [LEARNED], never change Temperature, top-k, top-p, penalties, beam width [CONFIG] (shape sampling) Capacity factor / dropless, expert-parallel degree, offloading \u0026amp; quantization policy [CONFIG] (MoE-specific serving) Logits, probabilities, KV cache [STATE] Per-token routing decision, dispatch/combine buffers, per-expert batch load [STATE] (MoE-specific) The course\u0026rsquo;s theme in one line: two deployments of identical MoE weights can differ in tone and determinism via the sampling knobs (same as dense) and in cost/latency via the serving knobs (offloading, quantization, expert-parallel layout) — same $\\theta$, different config. Parameters are the model; configuration is how you run it.\nConclusion MoE inference is the dense inference loop with everything in attention and sampling unchanged — same decoding strategies, same temperature/top-k/top-p, same KV cache (sparsity is in the FFN, not attention). The one decisive difference is the memory-vs-compute asymmetry: per-token FLOPs follow the small $P_{\\text{act}}$ ($\\approx 2P_{\\text{act}}$) but resident memory follows the large $P_{\\text{tot}}$, because data-dependent routing means every expert must stay loaded even though each token touches only $k/E$ of them. This makes MoE serving memory-capacity- and bandwidth-bound rather than compute-bound, reappears the training-time load-balance and capacity issues at batch time, and motivates MoE-specific serving — expert offloading, expert quantization, and expert-parallel inference — on top of the dense toolkit. An MoE serves at a small model\u0026rsquo;s speed and a large model\u0026rsquo;s memory: the inference-side face of the bargain struck in Lesson 1.\nAnd with that the arc is complete. We began by splitting one number into two — $P_{\\text{tot}}$ and $P_{\\text{act}}$ — and every lesson since has been a consequence of that split: the experts that grow $P_{\\text{tot}}$, the router that selects within it, the balancing and capacity machinery that makes selection work, the all-to-all that distributes it, the scaling law that justifies it, and now the serving asymmetry that is its final expression.\nThe whole course in one paragraph An MoE replaces each block\u0026rsquo;s dense FFN with $E$ experts and a tiny router $W_g$ that, per token, picks the top-$k$ and combines them, $\\mathrm{MoE}(\\mathbf x)=\\sum_{i\\in\\mathcal T_k}g_iE_i(\\mathbf x)$ (L2–L4), splitting one parameter count into total $P_{\\text{tot}}$ (memory) and activated $P_{\\text{act}}$ (compute) (L1). The router\u0026rsquo;s discrete top-$k$ has no useful gradient, so it trains only through the gate weights of its winners — which lets it collapse onto a few experts, countered by the auxiliary load-balancing loss $\\alpha E\\sum_i f_iP_i$, the z-loss, or bias control (L5). Fixed-shape hardware forces an expert capacity $C=\\lceil C_f kT/E\\rceil$ with token dropping (L6), and sharded experts make every MoE layer a pair of all-to-all collectives, rendering training communication-bound (L7). The payoff is a two-dimensional scaling trade-off — more experts lower loss at fixed FLOPs, converting memory into quality (L8) — at the cost of overfitting-prone fine-tuning that wants expert dropout, selective updates, and a frozen router (L9), and inference that runs at a small model\u0026rsquo;s compute but a large model\u0026rsquo;s memory (L10). One distinction carried us throughout and is worth keeping: all of $\\theta$ is stored and trained, but only a $k/E$ slice is computed per token — capacity and cost, finally pried apart.\n← Lesson 9 — Fine-Tuning MoE Models · Course index · Next: (none — final lesson)\nAppendices: A — MoE Family · B — Dense vs. MoE · C — Acronyms The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/tutorials/moe-lessons/lesson-10-inference-in-depth/","summary":"\u003ch1 id=\"lesson-10--inference-in-depth\"\u003eLesson 10 — Inference in Depth\u003c/h1\u003e\n\u003cp\u003e\u003cem\u003eTechnical course on MoE LLM mechanics — Lesson 10 of 10.\u003c/em\u003e\u003c/p\u003e\n\u003chr\u003e\n\u003ch2 id=\"recap-of-lesson-9\"\u003eRecap of Lesson 9\u003c/h2\u003e\n\u003cp\u003eAdapting an MoE uses the same SFT/RLHF/DPO machinery as a dense model, but sparsity makes it overfit (capacity is the large $P_{\\text{tot}}$ on small data, with uneven expert gradient coverage), countered by expert dropout, selective/LoRA fine-tuning, and the uniquely-MoE option of freezing the router. The output is one frozen, aligned $\\theta$. We now serve it. As in the companion course, \u003cstrong\u003enothing here updates a parameter; every knob is [CONFIG] and every buffer is [STATE]\u003c/strong\u003e — but the Lesson 1 memory-vs-compute asymmetry now becomes the central, practical fact of inference.\u003c/p\u003e","title":"Inference in Depth"},{"content":"Lesson 10 — Encoders in Practice \u0026amp; the Modern Landscape Technical course on encoder LLM mechanics — Lesson 10 of 10.\nRecap of Lesson 9 Fine-tuning an encoder means discarding the MLM head and attaching a typed task head (pooled classifier, per-token classifier, or span head), then training all of $\\theta$ on labeled data — the clean transfer-learning move that contrasts with the decoder\u0026rsquo;s generation-steering (SFT/RLHF/prompting). We have now built the encoder bottom to top. This final lesson zooms out: where encoders genuinely win in production, how the family evolved from BERT to ModernBERT, the encoder–decoder hybrids, and the decision of which to use.\n1. The killer application: embeddings and retrieval The single most important modern use of encoders is producing dense vector embeddings for search and retrieval. Pool an encoder\u0026rsquo;s output (Lesson 6) into one vector per text; similar meanings land nearby under cosine similarity; you can then search a corpus by nearest neighbors. This powers semantic search, deduplication, clustering, recommendation, and — most visibly — retrieval-augmented generation (RAG), where an encoder retrieves relevant documents that are then fed to a decoder to answer from.\nSentence-BERT (SBERT) is the key refinement. Raw BERT is poor at comparing sentences efficiently: scoring all pairs requires feeding each pair jointly through the model ($O(n^2)$ forward passes for $n$ sentences). SBERT fine-tunes BERT in a bi-encoder / Siamese setup — encode each sentence independently into a fixed vector, then compare vectors with cosine similarity — so embedding a corpus is $O(n)$ and search is a vector lookup. It trains with a contrastive or triplet objective that pulls paraphrases together and pushes unrelated sentences apart, and it uses mean pooling (Lesson 6) rather than [CLS].\nflowchart LR Q[\"query\"] --\u003e EQ[\"encoder → vector\"] D[\"documents\"] --\u003e ED[\"encoder → vectors (precomputed)\"] EQ --\u003e S[\"cosine similarity / ANN search\"] ED --\u003e S S --\u003e R[\"top-k docs\"] R --\u003e G[\"decoder LLM generates answer (RAG)\"] Contrast with the decoder. Here the two families are complementary, not competing. In a RAG system the encoder finds the right context (cheap, single-pass, bidirectional, embedding-native) and the decoder writes the answer (generative). \u0026ldquo;Encoder retrieves, decoder generates\u0026rdquo; is the dominant production pattern, and it uses each model for exactly what its architecture is good at.\nFigure 1. Encoder retrieves, decoder generates.\n2. Bi-encoders vs. cross-encoders Two ways to use an encoder for matching, with a speed/accuracy trade:\nBi-encoder (SBERT): encode query and document separately, compare vectors. Fast (documents pre-embedded once), scalable to millions of items, slightly less accurate because query and document never interact inside the model. Cross-encoder: feed query and document together as [CLS] query [SEP] doc [SEP] and output a relevance score from [CLS]. More accurate (full cross-attention between query and document) but you must run the model for every query-document pair — too slow to scan a corpus. The standard production pattern uses both: a bi-encoder retrieves a few hundred candidates fast, then a cross-encoder reranks the shortlist accurately. This two-stage retrieve-then-rerank pipeline is pure encoder territory.\nFigure 2. Retrieval is a bi-encoder\u0026rsquo;s job; reranking is a cross-encoder\u0026rsquo;s.\n3. The encoder family, in one pass A short tour (full detail in Appendix A). Each model is best understood as fixing a specific limitation from the earlier lessons.\nBERT (2018) — the original: bidirectional, MLM + NSP, WordPiece, learned-absolute positions ≤512. RoBERTa — same architecture, better training: drop NSP, dynamic masking, ~10× more data, bigger batches, longer. Showed BERT was badly under-trained (Lesson 8). ALBERT — parameter efficiency: factorized embeddings and cross-layer weight sharing shrink the model; replaces NSP with the harder Sentence-Order Prediction. DistilBERT — knowledge distillation: ~40% smaller, ~60% faster, ~97% of BERT\u0026rsquo;s quality, for deployment. ELECTRA — denser objective: replaced-token detection scores all positions (not just 15%), directly attacking the signal-density problem (Lesson 8) and matching BERT with far less compute. DeBERTa — better attention/positions: disentangled attention (separate content and relative-position vectors) and an enhanced mask decoder; long the top GLUE/SuperGLUE performer. ModernBERT (2024) — the modernized encoder: RoPE positions (breaking the 512 ceiling → 8192 context), pre-LN, GeGLU, FlashAttention, trained on far more (and code-inclusive) data. Brings a decade of decoder engineering back to the encoder. Contrast with the decoder. Notice the family\u0026rsquo;s direction: many of these advances (RoPE, pre-LN, gated MLPs, FlashAttention, \u0026ldquo;train longer on more data\u0026rdquo;) were pioneered on decoders and back-ported to encoders. The encoder line moved more slowly during the LLM boom because attention and money went to generation — ModernBERT is partly a catch-up that asks \u0026ldquo;what does a BERT look like with 2024 decoder tricks?\u0026rdquo;\n4. Encoder–decoder models: the bridge Some architectures use both. T5 and BART are full encoder–decoder Transformers: a bidirectional encoder reads the input, and an autoregressive decoder generates the output while cross-attending to the encoder\u0026rsquo;s representations. This is the original Transformer design (built for translation) and it shines on sequence-to-sequence tasks — summarization, translation, structured rewriting — where you need to read one text fully and write another.\nT5 frames every task as text-to-text (\u0026ldquo;translate English to German: …\u0026rdquo;, \u0026ldquo;summarize: …\u0026rdquo;) and pretrains with a span-corruption objective (mask contiguous spans, generate them) — an MLM-flavored objective adapted for a generative decoder. BART pretrains as a denoising autoencoder: corrupt the input (masking, deletion, shuffling) and reconstruct the original with the decoder. These sit between the pure families: bidirectional reading (encoder strength) plus generation (decoder strength), at the cost of roughly twice the machinery.\nContrast with the decoder. A decoder-only model folds \u0026ldquo;reading\u0026rdquo; and \u0026ldquo;writing\u0026rdquo; into one causal stack and conditions on input by putting it in the prompt. An encoder–decoder keeps them as separate modules joined by cross-attention. Decoder-only designs won the general-purpose-LLM race (simpler, one stack, KV-cache-friendly), but encoder–decoders remain strong where the input is long and fully available and the output is a transformation of it.\n5. Which should you reach for? A practical decision guide, summarizing the whole course:\nIf you need to… Use Why Classify, tag, score, or extract from given text Encoder bidirectional context, typed heads, one cheap forward pass, structured output Embed text for search / clustering / RAG retrieval Encoder (bi-encoder) embedding-native, single-pass, scales to millions Generate free-form text, chat, code, agents Decoder autoregressive generation is its entire purpose Do many tasks zero-/few-shot without labeled data Decoder in-context learning from a prompt Read a long input and write a transformed output Encoder–decoder (T5/BART) bidirectional read + generative write Build RAG Both encoder retrieves, decoder generates The headline: encoders for understanding, decoders for generation, encoder–decoders for transformation — and in real systems they cooperate far more than they compete. Encoders are not obsolete in the LLM era; they are the quiet, high-throughput workhorses behind search, moderation, classification, and the retrieval half of RAG, precisely because they are cheap, single-pass, and produce calibrated structured outputs that a generator cannot match economically.\nFigure 3. Which family to reach for.\nConclusion In practice, encoders dominate embeddings and retrieval: pooled into vectors (Sentence-BERT, mean pooling, bi-encoder training) they power semantic search, reranking, and the retrieval half of RAG, where they pair naturally with a generative decoder. The family evolved by fixing the course\u0026rsquo;s recurring limitations — RoBERTa fixed under-training, ELECTRA fixed signal density, DeBERTa improved attention/positions, ModernBERT broke the 512 ceiling with RoPE and imported modern decoder engineering — while encoder–decoders (T5, BART) bridge to generation for sequence-to-sequence work. The choice is rarely either/or: reach for an encoder to understand, a decoder to generate, an encoder–decoder to transform, and both to build RAG. That cooperative division of labor — and the architectural reasons behind it, traced from the single deleted causal mask of Lesson 3 — is the lasting takeaway of this course.\n← Previous: Lesson 9 — Fine-Tuning Encoders for Downstream Tasks · Course index · Next: (none — this is the final lesson)\nAppendices: A — Encoder Family · B — Encoder vs. Decoder Cheat Sheet · C — Acronyms The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/tutorials/encoder-deep-dive/lesson-10-practice-and-landscape/","summary":"\u003ch1 id=\"lesson-10--encoders-in-practice--the-modern-landscape\"\u003eLesson 10 — Encoders in Practice \u0026amp; the Modern Landscape\u003c/h1\u003e\n\u003cp\u003e\u003cem\u003eTechnical course on encoder LLM mechanics — Lesson 10 of 10.\u003c/em\u003e\u003c/p\u003e\n\u003chr\u003e\n\u003ch2 id=\"recap-of-lesson-9\"\u003eRecap of Lesson 9\u003c/h2\u003e\n\u003cp\u003eFine-tuning an encoder means discarding the MLM head and attaching a typed task head (pooled classifier, per-token classifier, or span head), then training all of $\\theta$ on labeled data — the clean transfer-learning move that contrasts with the decoder\u0026rsquo;s generation-steering (SFT/RLHF/prompting). We have now built the encoder bottom to top. This final lesson zooms out: where encoders genuinely win in production, how the family evolved from BERT to ModernBERT, the encoder–decoder hybrids, and the decision of which to use.\u003c/p\u003e","title":"Practice and Landscape"},{"content":"Chapter 10 — Raising a Helpful Assistant At the end of the last chapter we had a fully trained model — fluent, knowledgeable, and, surprisingly, not very useful. This chapter is about the gap between those two things, and how it gets closed. It is one of the most important and least understood stages in the whole story, because it is where a raw text-predictor becomes the polite, helpful, well-behaved assistant people actually talk to. The crucial thing to hold onto throughout is that no new kind of learning appears here. Every stage below is the same downhill walk from Chapter 9. All that changes is the teaching material.\nWhy the raw model isn\u0026rsquo;t helpful Imagine the result of all that training from Chapter 9: a model that has read a vast slice of the internet and become extremely good at predicting how text continues. This is called a base model, and it has a peculiar flaw. Ask it \u0026ldquo;What is the capital of France?\u0026rdquo; and it might not answer — it might instead continue with a list of more quiz questions, because on the web, a question like that often appears in exactly such a list. It is not being difficult; it is doing precisely what it was trained to do, which is to predict plausible continuations of text. It models how text behaves; it has not been taught to behave like an assistant.\nThe good news is that the helpful behaviors are already latent inside it — buried among everything else it absorbed, since the internet contains plenty of helpful answers too. They just need to be coaxed to the surface and made into the model\u0026rsquo;s default mode. That coaxing happens in two stages.\nFigure 1. Ask a base model the capital of France.\nStage one: showing it good answers The first stage is the most intuitive. You assemble a curated collection of example exchanges — a good question paired with an ideal answer, written or vetted by people — and you continue training the model on these polished demonstrations. It keeps doing exactly what it always did, predicting the next word, but now on examples of excellent, helpful responses rather than on the raw churn of the web. Importantly, it is graded only on the answer part of each example, not the question, so it learns the shape of being a good responder rather than how to ask questions.\nThis stage is often called supervised fine-tuning, and the analogy is an apprentice studying a portfolio of worked examples of excellent answers. Show enough of them and the model picks up the pattern: when someone asks something, you respond helpfully, directly, and in a consistent voice. This alone transforms the base model\u0026rsquo;s behavior — it now answers questions instead of echoing them.\nThere is a practical wrinkle worth mentioning, because you will hear its name. Adjusting all of a model\u0026rsquo;s billions of knobs for every bit of fine-tuning is expensive. A popular shortcut (called LoRA) freezes the original model entirely and learns only a small add-on adjustment — often less than one percent the size of the model — that snaps on top like a patch. It makes fine-tuning cheap and modular: you can keep one big frozen base model and maintain many small specialized patches for different purposes.\nStage two: learning from preferences Showing good answers gets you far, but it hits a ceiling. For most real questions there is no single \u0026ldquo;ideal answer\u0026rdquo; to demonstrate. How long should a good explanation be? How should the model handle a sensitive topic, or gracefully decline a harmful request, or admit it doesn\u0026rsquo;t know? You cannot easily write the one perfect demonstration of these things. But there is something humans can do reliably even when they can\u0026rsquo;t write the best answer themselves: look at two answers and say which one is better.\nThis observation powers the second stage. The model\u0026rsquo;s answers are shown to people in pairs, and the people simply pick the better of each pair. From a large pile of these comparisons, a separate model is trained — a reward model — whose entire job is to predict human preference, to act as a learned, automated stand-in for human taste. Once trained, it can score any answer the way a person would, millions of times over, without a human in the loop.\nThen the main model is tuned to produce answers that this reward model rates highly. Crucially, it is kept on a leash — not allowed to drift too far from the sensible language it already knew. The leash matters enormously, because without it the model discovers cheap tricks to game the scorer, producing strange text that the reward model loves but that reads to a human as gibberish. (This is a classic and instructive failure: optimize hard against any imperfect measure of \u0026ldquo;good,\u0026rdquo; and you get something that scores well and is bad.) With the leash holding it close to fluent language, this stage gives an assistant its characteristic feel — the polite tone, the sensible refusals, the calibrated hedging, the tendency to be genuinely helpful rather than merely plausible. This two-step dance of comparisons-then-tuning is usually called learning from human feedback.\nThere is a newer, simpler route that is increasingly popular, which skips building a separate reward model and the elaborate loop, and instead trains the main model directly on the human preference pairs — pushing it toward the answers people preferred and away from the ones they rejected. Same goal, less machinery. The details differ, but the spirit is identical: teach the model what people actually want, using comparisons rather than demonstrations.\nFigure 2. Comparisons, a stand-in for human taste, and a leash.\nThe same pattern beyond chatbots Although this two-stage story is told most often about chatbots, the underlying move — pretrain broadly, then adapt cheaply to a specific purpose — runs through the whole book. The reader models of Chapter 5 are adapted to specific tasks by attaching a small task-specific head and training briefly on a modest set of labeled examples, building on the general understanding they already absorbed. The mixture-of-experts models of Chapter 7 are fine-tuned with extra care, because their huge capacity makes them prone to over-memorizing small datasets — sometimes by freezing the router so a small dataset cannot tempt it back into using only a few experts. The translators of Chapter 6 are specialized on real pairs of source-and-target text. In every case the expensive, broad pretraining is done once, and adaptation is comparatively cheap.\nThe deepest contrast is between adapting a writer and adapting a reader. Adapting a writer is fundamentally a matter of behavior — coaxing a fluent generator into being helpful and well-mannered, which is what this whole chapter has described, and which can sometimes be achieved with nothing more than a cleverly worded prompt and no training at all. Adapting a reader is a matter of supervision — mapping its understanding onto specific labels, which always needs labeled examples. Steering generation versus mapping understanding: that is the cleanest summary of how the two great families are put to work.\nThe throughline It is worth saying once more, because it is so easily lost: across every stage in this chapter, there is no new learning machinery. Raw web text, then curated demonstrations, then human preference judgments — the teaching material changes, but the engine is always the same downhill walk from Chapter 9. And once all of this shaping is finished and the model ships, it freezes, exactly as Chapter 1 promised. Your conversations with it do not nudge a single knob. The helpfulness was pressed into the model during these stages, before you ever met it.\nFigure 3. One engine, three kinds of teaching material.\nWhat to carry forward A freshly trained model is a fluent predictor, not a helpful assistant — it models text rather than behaving like a respondent. It is raised into an assistant in two stages, both of them the ordinary learning of Chapter 9 pointed at different material. First it is shown curated examples of excellent answers, which teaches it the shape of being helpful. Then it is tuned against human preferences — usually via a learned stand-in for human taste, kept on a leash so it cannot game the score — which gives it its tone, its judgment, and its manners. The helpful behaviors were always latent in the base model; these stages bring them to the surface and make them the default. After that, the model is frozen and shipped.\nThe model is now both capable and helpful. All that remains is to understand what actually happens when you use it — the moment-to-moment business of generating an answer, and why that seemingly simple act is full of consequential choices and real costs. That is the final piece, in Chapter 11 .\n← Chapter 9 · How a Model Learns · Contents · Chapter 11 · How a Model Thinks in Real Time → The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/tutorials/inside-the-machines/chapter-10-raising-a-helpful-assistant/","summary":"\u003ch1 id=\"chapter-10--raising-a-helpful-assistant\"\u003eChapter 10 — Raising a Helpful Assistant\u003c/h1\u003e\n\u003cp\u003eAt the end of the last chapter we had a fully trained model — fluent, knowledgeable, and, surprisingly, not very useful. This chapter is about the gap between those two things, and how it gets closed. It is one of the most important and least understood stages in the whole story, because it is where a raw text-predictor becomes the polite, helpful, well-behaved assistant people actually talk to. The crucial thing to hold onto throughout is that \u003cem\u003eno new kind of learning appears here.\u003c/em\u003e Every stage below is the same downhill walk from Chapter 9. All that changes is the teaching material.\u003c/p\u003e","title":"Raising a Helpful Assistant"},{"content":"Appendix A — A Field Guide to the Encoder Family Technical course on encoder LLM mechanics — Appendix to Lessons 5 and 8.\nWhere this fits This is a side-quest off Lessons 5 (the MLM objective) and 8 (pretraining dynamics). The names that get thrown around — RoBERTa, ALBERT, ELECTRA, DeBERTa, ModernBERT — are all answers to a limitation that a specific lesson identified. Rather than catalogue them flatly, this appendix organizes the family by what each model changed, along four axes that the lessons already gave us: the objective (Lesson 5), the masking/training (Lesson 8), the position scheme (Lesson 2), and the parameter budget (Lessons 2, 4). Read any row as \u0026ldquo;BERT, but with this fixed.\u0026rdquo;\nA reading note on categories (Lesson 1\u0026rsquo;s framing): almost every choice below is [CONFIG] — architecture and training decided before the run. The weight matrices are [LEARNED]; the attention matrices and activations are [STATE]. When a model \u0026ldquo;is more efficient,\u0026rdquo; it is usually changing [CONFIG] (objective, masking, sharing) to extract more learning per FLOP, not inventing new kinds of numbers.\nThe baseline: BERT (2018) Everything else is defined relative to this. Bidirectional Transformer encoder (Lessons 3–4), MLM + NSP objective (Lesson 5), WordPiece vocabulary of 30,522 (Lesson 2), learned-absolute positions capped at 512 (Lesson 2), post-LN blocks with GELU (Lesson 4). Two sizes: base ($N=12$, $d=768$, ~110M params) and large ($N=24$, $d=1024$, ~340M). Its weaknesses, in the order the later models attack them: under-trained, NSP is useless, only 15% of positions supervised, learned-absolute positions cap length, and embeddings eat a big share of parameters.\nFigure 1. The baseline, and the five weaknesses the family attacks.\nRoBERTa — \u0026ldquo;BERT was under-trained\u0026rdquo; (training fix, Lesson 8) Same architecture, better recipe. RoBERTa changed only [CONFIG]: drop NSP entirely, use dynamic masking (fresh mask each epoch), train on ~10× more data (~160GB vs. ~16GB), much larger batches, longer schedules, and a byte-level BPE vocabulary. No architectural novelty — and it substantially beat BERT, proving the original was badly under-optimized. The lesson (8): for MLM\u0026rsquo;s low signal density, volume and training length are the cheapest large wins.\nALBERT — \u0026ldquo;shrink the parameters\u0026rdquo; (budget fix, Lessons 2, 4) Two parameter-reduction tricks plus a better objective. Factorized embedding parameterization: instead of a $V\\times d$ token embedding, use $V\\times e$ then $e\\times d$ with $e\\ll d$, decoupling vocabulary size from hidden width (recall from Lesson 2 that embeddings are a large share of BERT-base). Cross-layer parameter sharing: reuse the same block weights across all $N$ layers, drastically cutting parameter count. And it replaces NSP with Sentence-Order Prediction (SOP) — distinguish correctly-ordered adjacent sentences from swapped ones, a genuinely harder coherence task than NSP (Lesson 5). Trades compute for memory: fewer parameters, but sharing can slow convergence.\nDistilBERT — \u0026ldquo;make it deployable\u0026rdquo; (compression) Knowledge distillation: train a small student (6 layers) to match a frozen BERT teacher\u0026rsquo;s output distribution (plus MLM and a cosine-embedding loss). Result: ~40% fewer parameters, ~60% faster inference, ~97% of BERT\u0026rsquo;s GLUE. Pure deployment play — same objective family, smaller body. The encoder analogue of distilling any large model.\nELECTRA — \u0026ldquo;supervise every position\u0026rdquo; (objective fix, Lessons 5, 8) The most pointed attack on the 15% signal-density problem (Lesson 8). Instead of MLM, ELECTRA trains with replaced-token detection (RTD): a small generator replaces some tokens with plausible fakes, and the main encoder (the \u0026ldquo;discriminator\u0026rdquo;) classifies every token as original or replaced. Because the loss is defined at all positions, not just 15%, ELECTRA extracts far more signal per forward pass and reaches BERT-level quality with roughly a quarter of the compute. It is a GAN-flavored setup but trained with maximum likelihood, not adversarially.\n$$ \\mathcal L_{\\text{RTD}}=-\\sum_{i=1}^{n}\\Big[\\,\\mathbb 1[\\text{real}_i]\\log D_i+\\mathbb 1[\\text{fake}_i]\\log(1-D_i)\\Big],\\qquad D_i=\\sigma(\\mathbf w^\\top H^{(N)}_{i,:}). $$The discriminator is what you keep and fine-tune; the generator is discarded.\nFigure 2. Replaced-token detection: a loss at every position.\nDeBERTa — \u0026ldquo;better attention and positions\u0026rdquo; (mechanism fix, Lessons 2, 3) Two ideas. Disentangled attention: represent each token by two vectors — content and relative position — and compute attention scores from content-to-content, content-to-position, and position-to-content terms separately, rather than baking position into one summed input vector (contrast Lesson 2\u0026rsquo;s additive scheme). Enhanced mask decoder: re-inject absolute position information near the output layer for the MLM head. DeBERTa (and DeBERTaV3, which swaps MLM for ELECTRA-style RTD) topped GLUE/SuperGLUE for a long stretch and is a common high-accuracy default.\nModernBERT (2024) — \u0026ldquo;a BERT with modern decoder engineering\u0026rdquo; (everything, all lessons) The catch-up release. It imports the techniques the decoder world refined during the LLM boom: RoPE positions (Lesson 2) replacing learned-absolute, breaking the 512 ceiling to 8192 tokens; pre-LN blocks (Lesson 4) for stable training; GeGLU gated MLPs; FlashAttention and alternating local/global attention for speed; and training on far more data including code. The result is a drop-in encoder that is faster and far longer-context than BERT while retaining the bidirectional, MLM-trained, head-fine-tuned recipe of the whole course. It is the clearest evidence of the Lesson 10 point: most of these advances were decoder-pioneered and back-ported.\nThe family at a glance Model Changed vs. BERT Axis (lesson) One-line payoff BERT — (baseline) — bidirectional MLM encoder RoBERTa drop NSP, dynamic mask, 10× data, big batches training (8) BERT was under-trained ALBERT factorized embeds, layer sharing, SOP budget (2,4) far fewer parameters DistilBERT distillation to 6 layers compression 60% faster, 97% quality ELECTRA replaced-token detection (all positions) objective (5,8) dense signal, ¼ compute DeBERTa disentangled attention, rel. positions mechanism (2,3) top GLUE accuracy ModernBERT RoPE→8192, pre-LN, GeGLU, FlashAttn, more+code data all modernized long-context encoder Takeaway. The encoder family is not a random zoo: each member fixes a named limitation from the lessons — under-training (RoBERTa), parameter bloat (ALBERT), inference cost (DistilBERT), signal density (ELECTRA), the additive-position bottleneck (DeBERTa), and the 512 ceiling plus dated engineering (ModernBERT). Hold the baseline in mind and every model reads as a one-line diff.\nFigure 3. Each model is one line of diff against BERT.\n← Course index · Extends Lesson 5 — Masked Language Modeling and Lesson 8 — Pretraining Dynamics Appendices: A — Encoder Family · B — Encoder vs. Decoder Cheat Sheet · C — Acronyms The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/tutorials/encoder-deep-dive/appendix-a-encoder-family/","summary":"\u003ch1 id=\"appendix-a--a-field-guide-to-the-encoder-family\"\u003eAppendix A — A Field Guide to the Encoder Family\u003c/h1\u003e\n\u003cp\u003e\u003cem\u003eTechnical course on encoder LLM mechanics — Appendix to Lessons 5 and 8.\u003c/em\u003e\u003c/p\u003e\n\u003chr\u003e\n\u003ch2 id=\"where-this-fits\"\u003eWhere this fits\u003c/h2\u003e\n\u003cp\u003eThis is a side-quest off Lessons 5 (the MLM objective) and 8 (pretraining dynamics). The names that get thrown around — RoBERTa, ALBERT, ELECTRA, DeBERTa, ModernBERT — are all answers to a limitation that a specific lesson identified. Rather than catalogue them flatly, this appendix organizes the family by \u003cstrong\u003ewhat each model changed\u003c/strong\u003e, along four axes that the lessons already gave us: the \u003cstrong\u003eobjective\u003c/strong\u003e (Lesson 5), the \u003cstrong\u003emasking/training\u003c/strong\u003e (Lesson 8), the \u003cstrong\u003eposition scheme\u003c/strong\u003e (Lesson 2), and the \u003cstrong\u003eparameter budget\u003c/strong\u003e (Lessons 2, 4). Read any row as \u0026ldquo;BERT, but with this fixed.\u0026rdquo;\u003c/p\u003e","title":"Appendix A — A Field Guide to the Encoder Family"},{"content":"Appendix A — Acronyms \u0026amp; Abbreviations How to read this list. Every acronym and abbreviation that appears anywhere in the ten lessons is collected here, expanded in full, and explained in one or two sentences of plain language. The right-hand column points you to the lesson(s) where the term does real work, so you can jump back to its first proper use. Terms are grouped by theme rather than alphabetically, because related ideas are easier to learn together; an alphabetical quick-index follows at the end.\nCore model families Acronym Full form What it means Lesson(s) DDPM Denoising Diffusion Probabilistic Models The canonical diffusion model: a fixed forward noising chain plus a learned reverse denoiser, trained by noise-prediction regression. The backbone of the whole course. 1–9 DDIM Denoising Diffusion Implicit Models A non-Markovian reformulation of the same trained model that allows fast and even fully deterministic sampling in far fewer steps. 1, 3, 8, 9, 10 VAE Variational Autoencoder A latent-variable generative model with a learned encoder and decoder trained on the ELBO. A diffusion model is a VAE whose encoder is frozen to the noising chain. 1, 4, 7, 10 GAN Generative Adversarial Network A generator/discriminator pair trained as a min–max game; fast to sample but unstable. The main contrast class to diffusion. 1, 6, 8, 10 VDM Variational Diffusion Models A line of work expressing diffusion entirely through the (log-)signal-to-noise ratio, clarifying which design choices actually matter. 2, 4, 5, 7 LDM Latent Diffusion Models Diffusion run in the compressed latent space of an autoencoder rather than on raw pixels — the recipe behind Stable Diffusion. 10 NCSN Noise Conditional Score Network The score-matching predecessor to DDPM that learns the gradient of the log-density at many noise levels. 9 Objectives \u0026amp; training quantities Acronym Full form What it means Lesson(s) ELBO Evidence Lower Bound A tractable lower bound on the data log-likelihood; maximizing it is the principled training objective behind both VAEs and diffusion. 1, 4, 5, 7, 10 VLB Variational Lower Bound Same object as the ELBO, written $\\mathcal L_{\\text{vlb}}$; the per-step KL decomposition used as the full diffusion loss. 4, 7, 8 KL Kullback–Leibler divergence An asymmetric measure of how far one distribution is from another, written $D_{\\mathrm{KL}}(\\cdot\\,\\|\\,\\cdot)$. Each diffusion training term is a KL between the true and learned reverse step. 1, 4, 5, 7, 10 MSE Mean Squared Error The squared-distance regression loss; the simplified diffusion objective is just an MSE between true and predicted noise. 1, 5, 7, 8 MLE Maximum Likelihood Estimation Fitting parameters by maximizing the likelihood of the data; the gold standard that the ELBO approximates. 1 NLL Negative Log-Likelihood The quantity actually minimized (lower is better); the ELBO is an upper bound on the NLL. 4 SNR Signal-to-Noise Ratio The ratio $\\bar\\alpha_t/(1-\\bar\\alpha_t)$ of preserved signal to injected noise at step $t$; its logarithm is the natural axis for comparing schedules and loss weightings. 2, 3, 5, 7, 9 FID Fréchet Inception Distance A standard sample-quality metric comparing the statistics of generated and real images (lower is better). 5 LPIPS Learned Perceptual Image Patch Similarity A perceptual distance between images used to train the autoencoder in latent diffusion. 10 The continuous / score-based view Acronym Full form What it means Lesson(s) SDE Stochastic Differential Equation The continuous-time limit of the noising chain; diffusion is a particular variance-preserving SDE. 1, 2, 3, 8, 9, 10 ODE Ordinary Differential Equation The deterministic \u0026ldquo;probability-flow\u0026rdquo; companion to the SDE that shares its marginals — DDIM is its discretization. 1, 9, 10 VP Variance-Preserving The schedule family (used by DDPM) that keeps the marginal variance bounded near $1$ throughout noising. 2, 9 VP-SDE Variance-Preserving Stochastic Differential Equation The continuous-time SDE whose discretization is the DDPM forward chain. 2, 9 Samplers \u0026amp; solvers Acronym Full form What it means Lesson(s) DPM-Solver Diffusion Probabilistic Model Solver A fast, high-order ODE solver tailored to the diffusion ODE, enabling quality samples in ~10–20 steps. 9 DEIS Diffusion Exponential Integrator Sampler Another exponential-integrator fast solver for the diffusion ODE. 9 RK Runge–Kutta The classical family of ODE integrators (including Heun\u0026rsquo;s method) used as higher-order samplers. 9 EDM Elucidating the Design space of diffusion Models Karras et al.\u0026rsquo;s framework that unifies preconditioning, schedules, and weighting; source of the widely used $c_{\\text{skip}}/c_{\\text{out}}$ design. 5, 6, 7, 9 Architecture \u0026amp; conditioning Acronym Full form What it means Lesson(s) U-Net U-shaped (encoder–decoder) Network The image-to-image convolutional backbone with skip connections that implements the noise predictor $\\epsilon_\\theta$. 1, 5–10 DiT Diffusion Transformer A transformer-based alternative to the U-Net backbone that scales well at high resolution. 6, 9, 10 MLP Multi-Layer Perceptron A small fully connected network; used here to map the time embedding into per-block conditioning signals. 6, 7 FiLM Feature-wise Linear Modulation Conditioning by predicting a per-channel scale and shift applied to feature maps. 6 adaLN Adaptive Layer Normalization FiLM-style modulation applied to LayerNorm parameters; adaLN-Zero initializes it to the identity for stable training. 6, 10 adaGN Adaptive GroupNorm The GroupNorm analogue of adaLN, injecting time/conditioning as a scale-shift. 6 GN Group Normalization Normalization over groups of channels; the default in diffusion U-Nets because it is batch-size independent. 6 BN Batch Normalization Normalization over the batch; generally avoided in diffusion because its statistics interact badly with noise levels. 6, 7 SiLU Sigmoid Linear Unit (a.k.a. Swish) The smooth activation $x\\,\\sigma(x)$ used throughout modern diffusion backbones. 6 CFG Classifier-Free Guidance Steering generation toward a condition by extrapolating between conditional and unconditional noise predictions, with no separate classifier. 8, 9, 10 CLIP Contrastive Language–Image Pre-training A jointly trained text/image encoder; a common source of text embeddings for conditioning. 10 T5 Text-to-Text Transfer Transformer A large language model whose text encoder is used to condition image generation. 10 VQ Vector Quantization Discrete-codebook compression of the autoencoder latent in some latent-diffusion variants. 10 KL-VAE KL-regularized Variational Autoencoder The continuous-latent autoencoder (with a light KL penalty) used as the front end in latent diffusion. 10 Loss weighting \u0026amp; schedules Acronym Full form What it means Lesson(s) P2 Perception Prioritized (weighting) A loss weighting that emphasizes the noise levels most relevant to perceptual quality. 5 Min-SNR-γ Minimum Signal-to-Noise Ratio (gamma) A weighting that clips each step\u0026rsquo;s effective SNR at $\\gamma$, balancing easy and hard noise levels for faster convergence. 5, 7 Optimization \u0026amp; engineering Acronym Full form What it means Lesson(s) EMA Exponential Moving Average A slowly averaged copy of the weights used at sampling time; almost always improves diffusion sample quality. 1, 7, 8, 9 AdamW Adam with decoupled Weight decay The default optimizer for diffusion training. 7 WD Weight Decay $L_2$-style regularization on the weights (decoupled in AdamW). 7 AMP Automatic Mixed Precision Training/inference in mixed 16/32-bit precision for speed and memory savings. 7, 8 i.i.d. independent and identically distributed The standard assumption that data points are drawn independently from the same distribution. 7 GPU Graphics Processing Unit The accelerator hardware diffusion models are trained and sampled on. 10 Distillation \u0026amp; fast generation Acronym Full form What it means Lesson(s) SD / SD3 Stable Diffusion / Stable Diffusion 3 The widely used open latent-diffusion text-to-image model (and its third-generation, flow-matching version). 10 ADD Adversarial Diffusion Distillation A distillation method combining a diffusion objective with an adversarial loss for few-step generation. 10 LCM Latent Consistency Model A consistency-distillation approach enabling 1–4 step sampling in latent space. 10 Venues \u0026amp; organizations (references only) These appear only in citations and are listed for completeness: ICML (International Conference on Machine Learning), NeurIPS (Neural Information Processing Systems), ICLR (International Conference on Learning Representations), CVPR (Computer Vision and Pattern Recognition), ICCV (International Conference on Computer Vision), AAAI (Association for the Advancement of Artificial Intelligence), JASA (Journal of the American Statistical Association), MICCAI (Medical Image Computing and Computer-Assisted Intervention), and ACM (Association for Computing Machinery).\nNamed terms that are not acronyms. A few load-bearing names look like jargon but are people or concepts, not abbreviations: Tweedie\u0026rsquo;s formula, Feller\u0026rsquo;s condition, Anderson\u0026rsquo;s reverse-time SDE, Jensen\u0026rsquo;s inequality, Bayes\u0026rsquo; rule, Fokker–Planck equation, Euler–Maruyama and Heun integrators, Langevin dynamics, Lipschitz continuity, and slerp (spherical linear interpolation). They are explained where they appear in the lessons.\nAlphabetical quick-index AAAI · ACM · adaGN · adaLN · ADD · AdamW · AMP · BN · CFG · CLIP · CVPR · DDIM · DDPM · DEIS · DiT · DPM-Solver · EDM · ELBO · EMA · FID · FiLM · GAN · GN · GPU · ICCV · ICLR · ICML · i.i.d. · JASA · KL · KL-VAE · LCM · LDM · LPIPS · MICCAI · Min-SNR-γ · MLE · MLP · MSE · NCSN · NeurIPS · NLL · ODE · P2 · RK · SD/SD3 · SDE · SiLU · SNR · T5 · U-Net · VAE · VDM · VLB · VP · VP-SDE · VQ · WD\n← Course Index · Appendix B: Mathematical Formulas → ","permalink":"https://www.fanfani.net/tutorials/diffusion-lessons/appendix-a-acronyms/","summary":"\u003ch1 id=\"appendix-a--acronyms--abbreviations\"\u003eAppendix A — Acronyms \u0026amp; Abbreviations\u003c/h1\u003e\n\u003cblockquote\u003e\n\u003cp\u003e\u003cstrong\u003eHow to read this list.\u003c/strong\u003e Every acronym and abbreviation that appears anywhere in the ten lessons is collected here, expanded in full, and explained in one or two sentences of plain language. The right-hand column points you to the lesson(s) where the term does real work, so you can jump back to its first proper use. Terms are grouped by theme rather than alphabetically, because related ideas are easier to learn together; an alphabetical quick-index follows at the end.\u003c/p\u003e","title":"Appendix A — Acronyms"},{"content":"Appendix A — Acronyms \u0026amp; Abbreviations A glossary for the 10-lesson course. Each entry gives the expansion, a plain-language explanation, and where in the course it first matters. Grouped by theme so related terms sit together; use your viewer\u0026rsquo;s find (Ctrl/Cmd-F) to jump to a specific term.\nThis is a lookup table, not a chapter — keep it open beside the lessons and jump in when an unfamiliar acronym appears. The groups below trace the same arc the course does: the model\u0026rsquo;s components (Lesson 1–5), then the machinery that trains and aligns it (Lessons 6–9), then how it is run (Lesson 10). The symbol table at the end collects the single letters that recur as standard notation throughout.\n← Course index · Appendix B — Formulas Architecture \u0026amp; model components LLM — Large Language Model. A neural network trained to predict the next token in a sequence; the whole subject of the course. (Lessons 1, 2, 7, 8)\nFFN — Feed-Forward Network. The position-wise two-layer network inside each transformer block; applied independently to every token. Often used interchangeably with MLP in this context. (Lessons 1, 4)\nMLP — Multi-Layer Perceptron. A stack of fully-connected layers with a nonlinearity between them; in a transformer it is the FFN. (Lessons 1, 4, 7, 9)\nMHA — Multi-Head Attention. Running several attention \u0026ldquo;heads\u0026rdquo; in parallel, each on its own subspace, then concatenating and projecting the results so the model can attend to different patterns at once. (Lesson 3)\nGQA — Grouped-Query Attention. An inference-time saving where several query heads share one set of key/value heads, shrinking the KV cache with little quality loss. (Lessons 1, 10)\nMQA — Multi-Query Attention. The extreme of GQA: all query heads share a single key/value head, minimizing KV-cache memory. (Lessons 1, 10)\nKV cache — Key-Value Cache. A buffer that stores the key and value vectors already computed for past tokens, so generation doesn\u0026rsquo;t recompute them at every step. (Lessons 1, 10)\nNormalization \u0026amp; activations LayerNorm — Layer Normalization. Standardizes a token\u0026rsquo;s features (subtract mean, divide by standard deviation), then applies a learned scale and shift. Stabilizes training. (Lessons 1, 4)\nRMSNorm — Root-Mean-Square Normalization. A cheaper LayerNorm variant that divides by the root-mean-square of the features without subtracting the mean. (Lessons 1, 4)\nGELU — Gaussian Error Linear Unit. A smooth activation function, $z\\cdot\\Phi(z)$, where $\\Phi$ is the standard-normal CDF; a softer alternative to ReLU. (Lessons 1, 4)\nSwiGLU — Swish-Gated Linear Unit. A gated activation that multiplies a linear projection by a Swish/SiLU-gated one; common in modern transformer MLPs. (Lessons 1, 4)\nPositions \u0026amp; tokenization BPE — Byte-Pair Encoding. A subword tokenizer that repeatedly merges the most frequent adjacent symbol pair, building a vocabulary of common chunks. (Lessons 1, 2)\nWordPiece. A subword tokenizer that merges pairs by likelihood gain rather than raw frequency (used by BERT-family models). (Lesson 2)\nUnigram (SentencePiece). A tokenizer that starts from a large candidate vocabulary and prunes tokens by marginal likelihood. (Lesson 2)\nOOV — Out-Of-Vocabulary. A token or word the model\u0026rsquo;s vocabulary cannot represent directly; subword schemes largely eliminate this. (Lesson 2)\nRoPE — Rotary Position Embeddings. Encodes position by rotating query and key vectors by a position-dependent angle, so relative position falls out of the dot product. (Lessons 1, 2)\nPE — Positional Encoding. Any scheme (sinusoidal, learned, or rotary) that injects token-order information into the otherwise order-blind attention mechanism. (Lesson 2)\nTraining, loss \u0026amp; information CE — Cross-Entropy. The standard training loss: the average negative log-probability the model assigns to the true next token. (Lessons 5, 6, 9)\nKL — Kullback–Leibler divergence. A measure of how far the model\u0026rsquo;s distribution is from a target distribution; cross-entropy equals entropy plus KL. (Lessons 1, 6, 9, 10)\nPPL — Perplexity. The exponential of cross-entropy loss; reads as the effective number of equally-likely choices the model faces per token. (Lessons 1, 6)\nnats / bits. Units of information for entropy and loss: nats use the natural log, bits use log base 2 (bits double as a compression rate). (Lesson 6)\nMAP — Maximum A Posteriori. In decoding, the goal of finding the single highest-probability output sequence. (Lesson 10)\nOptimization \u0026amp; autodiff SGD — Stochastic Gradient Descent. Updating parameters opposite the gradient estimated from a random minibatch — the backbone of all training. (Lessons 1, 8)\nAdam — Adaptive Moment Estimation. An optimizer that keeps running averages of the gradient (first moment) and its square (second moment) to set a per-parameter step size. (Lessons 1, 8, 9)\nAdamW — Adam with decoupled Weight decay. Adam but with weight decay applied separately from the adaptive update; the standard choice for training transformers. (Lessons 1, 8)\nVJP — Vector-Jacobian Product. The core operation of reverse-mode autodiff (backpropagation): it pushes a gradient backward through one layer without ever forming the full Jacobian. (Lessons 1, 7)\nFLOP — Floating-Point Operation. One floating-point multiply or add; the unit used to measure compute cost (e.g. the $6PD$ training estimate). (Lesson 1)\nfp32 — 32-bit floating point. Single-precision numbers (4 bytes each); contrasted with lower-precision formats used to save memory at inference. (Lesson 1)\nFine-tuning \u0026amp; alignment SFT — Supervised Fine-Tuning. Continuing training on curated (prompt, response) demonstrations to shape the model\u0026rsquo;s behavior. (Lessons 1, 9)\nRLHF — Reinforcement Learning from Human Feedback. A two-stage post-training recipe: fit a reward model to human preferences, then optimize the policy against it. (Lessons 1, 9)\nLoRA — Low-Rank Adaptation. A parameter-efficient fine-tuning method that freezes the base weights and learns a small low-rank correction $\\tfrac{\\alpha}{r}BA$. (Lessons 1, 9)\nQLoRA — Quantized LoRA. LoRA on top of a base model quantized to 4-bit, cutting memory further so large models fine-tune on a single GPU. (Lesson 9)\nPPO — Proximal Policy Optimization. A reinforcement-learning algorithm that improves the policy using a clipped objective to prevent destabilizingly large updates. (Lessons 1, 9)\nDPO — Direct Preference Optimization. An alternative to RLHF that optimizes directly on preference pairs, skipping the separate reward model and RL loop. (Lessons 1, 9)\nRM — Reward Model. A model trained to score outputs by predicted human preference; the target that PPO optimizes against. (Lesson 9)\nNamed methods \u0026amp; concepts Bradley–Terry model. The preference model behind reward training: the probability one response beats another is $\\sigma$ of their reward difference. (Lessons 1, 9)\nChinchilla scaling laws. Empirical finding that compute-optimal training scales parameters and training tokens together in roughly equal proportion. (Lesson 8)\nFlashAttention. An IO-aware attention algorithm that computes attention in tiles, avoiding materializing the full $n\\times n$ score matrix. (Lesson 3)\nGoodhart\u0026rsquo;s Law. \u0026ldquo;When a measure becomes a target, it ceases to be a good measure\u0026rdquo; — invoked for reward over-optimization in RLHF. (Lesson 9)\nCommon single-letter symbols These aren\u0026rsquo;t acronyms but recur as standard notation across the lessons.\nSymbol Meaning $\\theta$ The full vector of learned parameters (the model itself). $P$ Total number of parameters. $D$ Number of training tokens. $V$ Vocabulary size. $N$ Number of transformer layers (depth). $d$ Model width / hidden dimension. $d_k,\\ d_v$ Key/query and value dimensions inside attention. $d_{ff}$ Hidden width of the MLP / feed-forward network. $h$ Number of attention heads. $n$ Sequence / context length. $\\eta$ Learning rate. $\\lambda$ Weight-decay coefficient. $\\hat{\\mathbf p}$ The model\u0026rsquo;s predicted next-token distribution. ← Course index · Appendix B — Formulas → · Appendix C — Attention Variants → ","permalink":"https://www.fanfani.net/tutorials/llm-lessons/appendix-a-acronyms/","summary":"\u003ch1 id=\"appendix-a--acronyms--abbreviations\"\u003eAppendix A — Acronyms \u0026amp; Abbreviations\u003c/h1\u003e\n\u003cp\u003e\u003cem\u003eA glossary for the 10-lesson course. Each entry gives the expansion, a plain-language explanation, and where in the course it first matters. Grouped by theme so related terms sit together; use your viewer\u0026rsquo;s find (Ctrl/Cmd-F) to jump to a specific term.\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eThis is a lookup table, not a chapter — keep it open beside the lessons and jump in when an unfamiliar acronym appears. The groups below trace the same arc the course does: the model\u0026rsquo;s components (Lesson 1–5), then the machinery that trains and aligns it (Lessons 6–9), then how it is run (Lesson 10). The symbol table at the end collects the single letters that recur as standard notation throughout.\u003c/em\u003e\u003c/p\u003e","title":"Appendix A — Acronyms"},{"content":"Appendix A — Formula Sheet Technical course on encoder–decoder transformers — reference appendix.\nA lookup table of every key equation, collected by lesson, with a one-line reminder of what each says. The cross-attention shapes (Lesson 4) are the ones worth memorizing. Keep this open alongside the lessons.\nLesson 1 — The seq2seq problem Conditional factorization — generate the output autoregressively, conditioned on the whole input:\n$$ P_\\theta(y\\mid x)=\\prod_{t=1}^{m} P_\\theta\\big(y_t\\mid y_{\\lt t},\\,x\\big). $$The two maps — encoder runs once, decoder runs per output token:\n$$ \\mathrm{Enc}: x\\mapsto H_{\\text{enc}}\\in\\mathbb R^{n\\times d},\\qquad \\mathrm{Dec}: (y_{\\lt t},H_{\\text{enc}})\\mapsto \\hat{\\mathbf p}_t\\in\\Delta^{V_y-1}. $$ Lesson 2 — Encoder (bidirectional self-attention) No causal mask (padding only):\n$$ \\mathrm{SelfAttn}_{\\text{bidir}}(H)=\\mathrm{softmax}\\!\\Big(\\tfrac{QK^\\top}{\\sqrt{d_k}}+M^{\\text{pad}}\\Big)V, \\quad Q,K,V=HW_Q,HW_K,HW_V,\\quad M^{\\text{enc}}_{ij}=0. $$Encoder block (two pre-norm residual sublayers):\n$$ H'=H+\\mathrm{MHA}_{\\text{bidir}}(\\mathrm{Norm}(H)),\\qquad H^{\\text{out}}=H'+\\mathrm{FFN}(\\mathrm{Norm}(H')). $$ Lesson 3 — Decoder (causal self-attention) Causal mask:\n$$ M^{\\text{causal}}_{ij}=\\begin{cases}0,\u0026 j\\le i\\\\ -\\infty,\u0026 j\\gt i\\end{cases} $$Decoder block (three pre-norm residual sublayers):\n$$ \\begin{aligned} H_1 \u0026= H + \\mathrm{SelfAttn}_{\\text{causal}}(\\mathrm{Norm}(H))\\\\ H_2 \u0026= H_1 + \\mathrm{CrossAttn}(\\mathrm{Norm}(H_1),H_{\\text{enc}})\\\\ H_3 \u0026= H_2 + \\mathrm{FFN}(\\mathrm{Norm}(H_2)) \\end{aligned} $$Output head: $\\ \\mathbf z_t=\\mathbf h_t^\\top W_U,\\quad \\hat{\\mathbf p}_t=\\mathrm{softmax}(\\mathbf z_t),\\quad W_U=E^\\top$ (tied).\nLesson 4 — Cross-attention (the only new mechanism) Two sources — queries from decoder, keys/values from encoder:\n$$ Q=H_{\\text{dec}}W_Q^c\\in\\mathbb R^{m\\times d_k},\\quad K=H_{\\text{enc}}W_K^c\\in\\mathbb R^{n\\times d_k},\\quad V=H_{\\text{enc}}W_V^c\\in\\mathbb R^{n\\times d_v}. $$Rectangular attention ($m$ output rows, $n$ input columns), source padding mask, no causal mask:\n$$ A=\\mathrm{softmax}\\!\\Big(\\tfrac{QK^\\top}{\\sqrt{d_k}}+M^{\\text{pad}}\\Big)\\in\\mathbb R^{m\\times n},\\qquad \\mathrm{CrossAttn}=AV\\in\\mathbb R^{m\\times d_v}. $$$$ \\mathbf o_i=\\sum_{j=1}^{n}A_{ij}\\mathbf v_j,\\qquad \\textstyle\\sum_{j=1}^{n}A_{ij}=1\\ \\ \\forall i. $$Gradient into the encoder (the learning bridge):\n$$ \\frac{\\partial\\mathcal L}{\\partial H_{\\text{enc}}}=\\frac{\\partial\\mathcal L}{\\partial K}\\frac{\\partial K}{\\partial H_{\\text{enc}}}+\\frac{\\partial\\mathcal L}{\\partial V}\\frac{\\partial V}{\\partial H_{\\text{enc}}}. $$ Lesson 5 — Full architecture \u0026amp; parameter budget $$ P\\approx 12\\,N_{\\text{enc}}\\,d^2 + 16\\,N_{\\text{dec}}\\,d^2 + Vd, \\quad\\text{(enc block }12d^2,\\ \\text{dec block }16d^2,\\ \\text{cross-attn }+4d^2). $$T5 relative-position bias (added to logits, no input position embedding):\n$$ s_{ij}=\\frac{\\mathbf q_i^\\top\\mathbf k_j}{\\sqrt{d_k}}+b_{\\,\\text{bucket}(i-j)}. $$ Lesson 6 — Prefix-LM mask $$ M^{\\text{prefix}}_{ij}=\\begin{cases} 0,\u0026 j\\le n\\ (\\text{bidirectional prefix})\\\\ 0,\u0026 n\\lt j\\le i\\ (\\text{causal suffix, past})\\\\ -\\infty,\u0026 j\\gt i,\\ j\\gt n\\ (\\text{future suffix}) \\end{cases} $$ Lesson 7 — Denoising objectives Span corruption (T5): spans → sentinels in $\\tilde x$; decoder emits the missing spans. BART: mask / delete / infill / permute sentences / rotate document; decoder rebuilds the whole document.\nLoss (same for both, and for fine-tuning):\n$$ \\mathcal L(\\theta)=-\\frac{1}{|y|}\\sum_{t}\\log P_\\theta(y_t\\mid y_{\\lt t},\\tilde x). $$ Lesson 8 — Teacher forcing Shifted decoder input / targets:\n$$ \\text{input}=(\\langle\\text{bos}\\rangle,y_1,\\dots,y_{m-1}),\\qquad \\text{targets}=(y_1,\\dots,y_m). $$Logit gradient (predicted − target):\n$$ \\frac{\\partial\\mathcal L}{\\partial\\mathbf z_t}=\\hat{\\mathbf p}_t-\\mathbf y_t. $$ Lesson 9 — Inference Cross-attention KV cache — computed once from $H_{\\text{enc}}$, fixed for all steps; self-attention cache grows by one per step.\nLength-normalized beam score:\n$$ \\text{score}(y)=\\frac{1}{\\mathrm{lp}(m)}\\sum_{t=1}^{m}\\log\\hat p_{y_t},\\qquad \\mathrm{lp}(m)=\\frac{(5+m)^\\alpha}{6^\\alpha}. $$Coverage vector / loss:\n$$ \\mathbf c^t=\\sum_{t'\\lt t}A_{t',:},\\qquad \\mathcal L_{\\text{cov}}=\\sum_t\\sum_j \\min(A_{tj},c^t_j). $$Pointer-generator copy:\n$$ P(w)=p_{\\text{gen}}P_{\\text{vocab}}(w)+(1-p_{\\text{gen}})\\!\\!\\sum_{j:x_j=w}\\!\\!A_{tj},\\qquad p_{\\text{gen}}\\in[0,1]. $$ Lesson 10 — Evaluation \u0026amp; distillation BLEU (precision + brevity penalty):\n$$ \\text{BLEU}=\\min(1,e^{1-r/c})\\cdot\\exp\\!\\Big(\\textstyle\\sum_{n=1}^{4}w_n\\log p_n\\Big). $$Distillation loss (hard labels + softened teacher):\n$$ \\mathcal L_{\\text{KD}}=(1-\\lambda)\\,\\mathrm{CE}(y,\\hat{\\mathbf p}^{\\,s})+\\lambda T^2 D_{\\mathrm{KL}}\\!\\big(\\hat{\\mathbf p}^{\\,t}_T\\|\\hat{\\mathbf p}^{\\,s}_T\\big). $$ ← Course index · Appendix B — Architecture Comparison · Appendix C — Acronyms ","permalink":"https://www.fanfani.net/tutorials/encoder-decoder-lessons/appendix-a-formulas/","summary":"\u003ch1 id=\"appendix-a--formula-sheet\"\u003eAppendix A — Formula Sheet\u003c/h1\u003e\n\u003cp\u003e\u003cem\u003eTechnical course on encoder–decoder transformers — reference appendix.\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003eA lookup table of every key equation, collected by lesson, with a one-line reminder of what each says. The cross-attention shapes (Lesson 4) are the ones worth memorizing. Keep this open alongside the lessons.\u003c/p\u003e\n\u003chr\u003e\n\u003ch2 id=\"lesson-1--the-seq2seq-problem\"\u003eLesson 1 — The seq2seq problem\u003c/h2\u003e\n\u003cp\u003e\u003cstrong\u003eConditional factorization\u003c/strong\u003e — generate the output autoregressively, conditioned on the whole input:\u003c/p\u003e\n$$\nP_\\theta(y\\mid x)=\\prod_{t=1}^{m} P_\\theta\\big(y_t\\mid y_{\\lt t},\\,x\\big).\n$$\u003cp\u003e\u003cstrong\u003eThe two maps\u003c/strong\u003e — encoder runs once, decoder runs per output token:\u003c/p\u003e","title":"Appendix A — Formula Sheet"},{"content":"Chapter 11 — How a Model Thinks in Real Time We have built a model (Chapter 9) and raised it into something helpful (Chapter 10). Now we watch it speak. This final chapter of the lifecycle is about inference — the moment-to-moment business of producing an answer from a frozen model. It is tempting to assume this is the simple part, the mere playing-back of what was learned. It is not. The act of generating is full of consequential choices that change the model\u0026rsquo;s whole character, and it carries real costs that shape how these systems are built and served. Understanding inference is what separates a vague sense of \u0026ldquo;the AI answers\u0026rdquo; from a real grasp of what is happening.\nGeneration is a loop of choices Recall from Chapter 4 that a storyteller does not output a word. It outputs odds — a ranked set of probabilities across all the words it knows. Generation, then, is a loop: produce the odds for the next word, pick one, glue it on, and repeat with the now-longer text. All the interesting behavior lives in that one picking step, and none of it changes the model. Two copies of the exact same frozen model can feel like completely different personalities depending only on how that pick is made.\nThe simplest policy is to always take the single most likely word. It sounds safe, and for a factual lookup it often is, but it tends to produce dull, repetitive text and can get the model stuck in loops, saying the same phrase over and over. The livelier alternative is to sample — to roll dice weighted by the odds, so the most likely word usually wins but sometimes a less likely one slips through. This brings variety and a spark of life, at the occasional cost of picking something odd from the long tail of unlikely words.\nThe master dial over all this is called temperature, and it is worth knowing because it is exactly the knob that makes a model feel \u0026ldquo;creative\u0026rdquo; or \u0026ldquo;boring.\u0026rdquo; Turn the temperature down and the model becomes conservative and predictable, strongly favoring its top choice — ideal for factual answers, code, anything where you want the safe option. Turn it up and the odds flatten out, making the model adventurous and surprising, eventually tipping into incoherence if pushed too far — useful for brainstorming and fiction. Same model, different personality, set by a single dial. Alongside it sit sensible guardrails that, before rolling the dice, throw away the obviously bad long-tail options — keeping only the most plausible handful, and adapting how many that is to how confident the model is at each step (few when it is sure, many when it is genuinely uncertain). The headline point is the one to remember: the model is fixed; how you run it is a choice, and that choice is a large part of what users experience as the model\u0026rsquo;s character.\nFigure 1. Three ways to pick the next word, and one master dial.\nWhy long conversations get expensive — and the fix There is a cost lurking in this loop that explains a great deal about how these systems behave. Naively, to produce each new word, the model would have to re-read the entire conversation so far from scratch. As the conversation grows, that becomes brutally wasteful — re-reading thousands of words just to add one more, then re-reading thousands-plus-one to add the next.\nThe fix is to cache the model\u0026rsquo;s work on all the earlier words, so that each new word only requires processing that one new word against the saved history rather than redoing everything. (This saved working memory is often called the KV cache.) This single trick is the difference between long conversations being feasible and being impossible. But it has a catch that ripples through the whole field: the cache grows with the length of the conversation, steadily eating memory. This — not the size of the model itself — is the main reason very long contexts are expensive. Much of the engineering behind \u0026ldquo;this model can handle a whole book\u0026rdquo; is really clever management of this ballooning cache, including tricks that let different parts of the model share cached information to shrink it with little loss of quality. Other speed tricks ride alongside: a small fast model can draft several words at once and let the big model check them all in a single pass, and the model\u0026rsquo;s numbers can be stored in a more compact, lower-precision form so they move through memory faster.\nFigure 2. What the saved working memory removes, and what it costs instead.\nWhen the bottleneck is memory, not thinking The mixture-of-experts models of Chapter 7 deserve a return visit here, because they invert the usual picture of what makes a model expensive to run. An ordinary dense model is limited mainly by raw computation — how much arithmetic it must do per word. A mixture-of-experts model does only a little arithmetic per word, since it activates just a few of its experts — but it is limited instead by memory, because all of its experts must be kept loaded and on hand even though any given word uses only a couple of them. You cannot keep just the popular experts loaded and discard the rest, because the router decides afresh for each word, so over a real stretch of text every expert eventually gets called. The whole hospital must stay staffed even though each patient sees only two doctors.\nThis is why serving these models is its own discipline, aimed squarely at the memory problem rather than the compute problem: keeping the busiest experts in fast memory while fetching colder ones on demand, shrinking each expert into a more compact form, and spreading the experts across many machines. It is the final, practical face of the bargain struck back in Chapter 7 — a mixture-of-experts model spends memory to deliver a large model\u0026rsquo;s quality at a small model\u0026rsquo;s running speed.\nImage generation has its own clock Diffusion models, from Chapter 8, run on a completely different rhythm, and it is worth placing it beside the language case. A language model\u0026rsquo;s generation cost grows with the length of what it writes — each word an added turn of the loop. A diffusion model\u0026rsquo;s cost is instead dominated by the number of refinement steps it takes to clarify the image from static. Early methods needed hundreds or a thousand such steps, making image generation slow; the central thrust of progress, as we saw, has been to cut that number dramatically — to a few dozen steps, or even a handful — by descending the noise ladder in larger leaps and by working in a compressed space rather than on raw pixels. Where a chatbot\u0026rsquo;s running cost is about managing a growing cache as the text lengthens, an image generator\u0026rsquo;s is about minimizing the number of passes needed to develop the picture. Same underlying concern — do less work to produce the output — taking two quite different forms.\nFigure 3. Three architectures, three different bottlenecks.\nWhat to carry forward Inference is not mere playback. Generating an answer is a loop of choices, governed by dials — chiefly temperature — that can make one frozen model feel terse and factual or wild and creative, without changing a single knob inside it. The loop carries a real and growing cost, tamed by caching the model\u0026rsquo;s work so each new word is cheap, with the size of that cache, not the model itself, being the main reason long contexts are expensive. Different architectures bottleneck differently: dense models on computation, mixture-of-experts models on memory, diffusion models on the number of refinement steps. In every case, the engineering of running these systems is a continual effort to do less work for each piece of output.\nThat completes the lifecycle: built, raised, and run. We have now seen every major piece — the foundations, the four language architectures, the image-sculptor, and the full arc from training to deployment. The closing chapter steps back from all of it to show how the pieces fit into one picture, and how to think about which tool suits which job. Turn to Chapter 12 .\n← Chapter 10 · Raising a Helpful Assistant · Contents · Chapter 12 · The Whole Landscape → The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/tutorials/inside-the-machines/chapter-11-how-models-run/","summary":"\u003ch1 id=\"chapter-11--how-a-model-thinks-in-real-time\"\u003eChapter 11 — How a Model Thinks in Real Time\u003c/h1\u003e\n\u003cp\u003eWe have built a model (Chapter 9) and raised it into something helpful (Chapter 10). Now we watch it speak. This final chapter of the lifecycle is about \u003cem\u003einference\u003c/em\u003e — the moment-to-moment business of producing an answer from a frozen model. It is tempting to assume this is the simple part, the mere playing-back of what was learned. It is not. The act of generating is full of consequential choices that change the model\u0026rsquo;s whole character, and it carries real costs that shape how these systems are built and served. Understanding inference is what separates a vague sense of \u0026ldquo;the AI answers\u0026rdquo; from a real grasp of what is happening.\u003c/p\u003e","title":"How a Model Thinks in Real Time"},{"content":"Appendix A — The Chinchilla Law: Compute-Optimal Scaling Technical course on LLM mechanics — Appendix to Lesson 8.\nWhere this fits This is a side-quest off Lesson 8. There, scaling laws appeared as the planning layer of a training run — the rule that tells you, before you spend a dollar of compute, how big to build and how much to feed. This appendix follows that thread to its practical conclusion: the Chinchilla recipe. Lesson 8 §7 introduced scaling laws this way: pretraining loss falls as a smooth power law in parameters and data,\n$$ \\mathcal L(N,D)\\approx \\mathcal L_\\infty + \\Big(\\tfrac{A}{N}\\Big)^{\\alpha}+\\Big(\\tfrac{B}{D}\\Big)^{\\beta}, $$down to an irreducible floor $\\mathcal L_\\infty$ (the entropy of language, Lesson 6). Here $N$ is the number of model [LEARNED] parameters and $D$ is the number of training tokens. This appendix derives the practical consequence — the Chinchilla compute-optimal rule (Hoffmann et al., 2022, \u0026ldquo;Training Compute-Optimal Large Language Models\u0026rdquo;) — that tells you, for a fixed compute budget, how big to make the model and how much data to feed it. Both $N$ and $D$ are [CONFIG]: chosen before the run, not learned by it.\n1. The one constraint that makes the problem well-posed From Lesson 1, the dominant training cost is\n$$ C \\approx 6\\,N\\,D \\quad\\text{(FLOPs)}, $$where the factor 6 counts the forward pass ($\\approx 2ND$) plus the backward pass ($\\approx 4ND$) per token. This is the budget line. The question is not \u0026ldquo;how low can loss go?\u0026rdquo; — with infinite compute, push both $N$ and $D$ up. The real-world question is constrained:\n$$ \\min_{N,\\,D}\\ \\mathcal L(N,D)\\quad\\text{subject to}\\quad 6ND = C . $$Spend on a bigger model and you have fewer tokens to train it on; spend on more tokens and the model must be smaller. There is an interior optimum, and finding it is the whole game.\n2. Kaplan vs. Chinchilla — the correction that mattered The earlier Kaplan et al. (2020) laws concluded that, given more compute, you should spend most of it on model size and comparatively little extra on data — roughly $N\\propto C^{0.73}$, $D\\propto C^{0.27}$. Under that guidance the flagship models of 2020–2021 were large but under-trained: GPT-3 had $N\\approx1.75\\times10^{11}$ parameters trained on only $D\\approx3\\times10^{11}$ tokens — a ratio of under 2 tokens per parameter.\nChinchilla re-ran the experiment with a cleaner methodology — crucially, tuning the learning-rate schedule to match each token budget (Lesson 8\u0026rsquo;s cosine decay should end as training ends; Kaplan\u0026rsquo;s runs decayed on a fixed schedule that disadvantaged the data-heavy points). The corrected conclusion: $N$ and $D$ should grow at nearly equal rates with compute,\n$$ N_{\\text{opt}}\\propto C^{\\,a},\\qquad D_{\\text{opt}}\\propto C^{\\,b},\\qquad a\\approx b\\approx 0.5 . $$The headline demonstration: Chinchilla (70B params, 1.4T tokens) outperformed Gopher (280B params, 300B tokens) using the same compute budget — a 4× smaller model beating a 4× larger one, purely by rebalancing toward data.\nFigure 1. How to split a compute budget, before and after the correction.\n3. Deriving the optimum Take the loss model from §0 and drop the constant floor (it doesn\u0026rsquo;t affect the argmin):\n$$ \\mathcal L(N,D)= \\frac{A}{N^{\\alpha}}+\\frac{B}{D^{\\beta}} . $$Hoffmann et al. fit $\\alpha\\approx0.34$, $\\beta\\approx0.28$ (with $A\\approx406$, $B\\approx410$, $\\mathcal L_\\infty\\approx1.69$ nats). Now substitute the budget constraint $D = C/(6N)$ to get a one-variable problem:\n$$ \\mathcal L(N)= \\frac{A}{N^{\\alpha}}+B\\Big(\\frac{6N}{C}\\Big)^{\\beta}. $$Differentiate and set to zero:\n$$ \\frac{d\\mathcal L}{dN}= -\\alpha\\frac{A}{N^{\\alpha+1}}+\\beta B\\Big(\\frac{6}{C}\\Big)^{\\beta}N^{\\beta-1}=0 . $$Rearranging gives $N^{\\alpha+\\beta}\\propto C^{\\beta}$, i.e.\n$$ \\boxed{\\,N_{\\text{opt}}\\propto C^{\\,\\beta/(\\alpha+\\beta)},\\qquad D_{\\text{opt}}\\propto C^{\\,\\alpha/(\\alpha+\\beta)}\\,.} $$The two exponents sum to 1 (they must, since $C\\propto ND$). Plugging the fitted $\\alpha\\approx0.34,\\ \\beta\\approx0.28$ gives exponents $\\tfrac{\\beta}{\\alpha+\\beta}\\approx0.45$ and $\\tfrac{\\alpha}{\\alpha+\\beta}\\approx0.55$ — close to the symmetric $0.5/0.5$ headline, and worlds apart from Kaplan\u0026rsquo;s $0.73/0.27$. The key qualitative fact is that $\\alpha\\approx\\beta$: the loss is roughly equally sensitive to a fractional shortfall in parameters or in data, so a balanced split is optimal.\n4. The rule of thumb: ~20 tokens per parameter Because $N_{\\text{opt}}$ and $D_{\\text{opt}}$ scale with nearly the same power of $C$, their ratio is approximately constant across budgets. Chinchilla\u0026rsquo;s fits put it at\n$$ \\frac{D_{\\text{opt}}}{N_{\\text{opt}}}\\approx 20\\ \\text{tokens per parameter}, $$the single number most people remember. It makes planning a one-line calculation. Given a compute budget $C$:\n$N_{\\text{opt}}=\\sqrt{C/(6\\cdot20)}$ and $D_{\\text{opt}}=20\\,N_{\\text{opt}}$ (from $C=6ND$ with $D=20N$). Read off architecture (depth/width/heads, Lesson 4) to hit $N_{\\text{opt}}$, and assemble $D_{\\text{opt}}$ tokens of data. Compute $C$ (FLOPs) $N_{\\text{opt}}$ (params) $D_{\\text{opt}}$ (tokens) $1\\times10^{21}$ $\\approx 2.9$ B $\\approx 58$ B $1\\times10^{22}$ $\\approx 9.1$ B $\\approx 183$ B $1\\times10^{23}$ $\\approx 29$ B $\\approx 577$ B $5.76\\times10^{23}$ $\\approx 69$ B (Chinchilla) $\\approx 1.4$ T The last row reproduces the actual Chinchilla design point as a sanity check: $6\\cdot(70\\times10^9)\\cdot(1.4\\times10^{12})\\approx5.9\\times10^{23}$ FLOPs.\nFigure 2. The planning table: one number, four budgets.\n5. Why \u0026ldquo;compute-optimal\u0026rdquo; is not \u0026ldquo;deployment-optimal\u0026rdquo; So far the rule has been airtight on its own terms. The catch is in those terms. Chinchilla minimizes training loss for a fixed training budget. It says nothing about inference, and that gap drives most modern practice. Once a model is deployed it is run on billions of tokens, each forward pass costing $\\approx2N$ FLOPs (Lesson 1) and $\\approx2N$ bytes of weight reads. A smaller $N$ is cheaper forever. So if you will serve the model heavily, it is rational to deliberately under-size $N$ and over-train on data well past the 20× point — spending more training compute than Chinchilla prescribes to buy a permanently cheaper, smaller model at the same quality.\nThis is exactly the regime of models like LLaMA: 7B parameters trained on ~1–2T+ tokens (well over 100 tokens/param), far past compute-optimal for training, but excellent on a per-inference-FLOP basis. The Chinchilla rule is the right default when training compute dominates total lifetime compute; it is the wrong default when inference does.\nTwo further caveats keep the law honest:\nData quality and repetition. The law assumes fresh, high-quality tokens. Repeating data has sharply diminishing (eventually negative) returns, and the finite supply of high-quality text is now a binding constraint that the original fits did not model. It predicts pretraining loss, not capability. Cross-entropy in nats is a smooth proxy; downstream task performance can move in steps (emergence) that the smooth curve does not directly forecast. Figure 3. The rule is airtight on its own terms — and those terms exclude inference.\nConclusion The Chinchilla law turns a vague \u0026ldquo;make it bigger\u0026rdquo; intuition into a constrained optimization: minimize $\\mathcal L(N,D)=A/N^{\\alpha}+B/D^{\\beta}$ subject to the budget line $C=6ND$. Because the fitted sensitivities are comparable ($\\alpha\\approx\\beta$), the optimum splits compute almost evenly between model and data — $N_{\\text{opt}}\\propto C^{0.5}$, $D_{\\text{opt}}\\propto C^{0.5}$ — collapsing to the memorable ~20 tokens per parameter. This corrected the Kaplan-era bias toward enormous, under-trained models (Chinchilla-70B beating Gopher-280B at equal compute being the proof). But \u0026ldquo;compute-optimal for training\u0026rdquo; is not \u0026ldquo;optimal for serving\u0026rdquo;: when a model\u0026rsquo;s lifetime is dominated by inference, the right move is to under-size $N$ and over-train on data, accepting a worse training-compute trade for a permanently cheaper deployed model. The law is a planning instrument — [CONFIG] chosen before a run — not a law of nature, and its assumptions about data quality and the loss-to-capability mapping are where it bends.\n← Lesson 8 — Optimizers, Learning Rate \u0026amp; Batches · Course index · Lesson 10 — Inference in Depth → The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/tutorials/llm-lessons/appendix-a-chinchilla-scaling-laws/","summary":"\u003ch1 id=\"appendix-a--the-chinchilla-law-compute-optimal-scaling\"\u003eAppendix A — The Chinchilla Law: Compute-Optimal Scaling\u003c/h1\u003e\n\u003cp\u003e\u003cem\u003eTechnical course on LLM mechanics — Appendix to Lesson 8.\u003c/em\u003e\u003c/p\u003e\n\u003chr\u003e\n\u003ch2 id=\"where-this-fits\"\u003eWhere this fits\u003c/h2\u003e\n\u003cp\u003eThis is a side-quest off Lesson 8. There, scaling laws appeared as the \u003cstrong\u003eplanning layer\u003c/strong\u003e of a training run — the rule that tells you, before you spend a dollar of compute, how big to build and how much to feed. This appendix follows that thread to its practical conclusion: the Chinchilla recipe. Lesson 8 §7 introduced scaling laws this way: pretraining loss falls as a smooth power law in parameters and data,\u003c/p\u003e","title":"Appendix A — Chinchilla Scaling Laws"},{"content":"Appendix B — Architecture Comparison \u0026amp; Model Zoo Technical course on encoder–decoder transformers — reference appendix.\nWhere this fits This appendix is the side-by-side that Lessons 1 and 6 build toward. The three transformer families are not three different inventions — they are the same machinery under different attention masks (and different sources for keys/values). This sheet lays them on one table along the axes that actually distinguish them, then names the real models so the abstractions have faces. Read it after Lesson 6, or keep it open from the start as a map.\n1. The three families on four axes Axis Encoder-only Decoder-only Encoder–decoder Attention mask bidirectional ($M=0$) causal ($M$ lower-tri) bidirectional encoder + causal decoder Key/value source self self self (within each stack) + cross (decoder reads encoder) New mechanism — — cross-attention Conditioning on input n/a (no generation) implicit: input in the causal prefix explicit: separate stack + cross-attention every decoder layer Input read bidirectionally causally (left-to-right) bidirectionally (by the encoder) Generates output? no — produces representations yes (continue the stream) yes (decoder, conditioned) Pretraining objective MLM (mask \u0026amp; predict in place) next-token CE denoising / span corruption Parameters per layer $12d^2$ $12d^2$ $12d^2$ (enc) / $16d^2$ (dec) Output lengths = input length one stream input $n$, output $m$, decoupled Natural tasks classification, NER, retrieval, embeddings open-ended generation, chat, few-shot translation, summarization, structured transduction The progression left-to-right across \u0026ldquo;attention mask\u0026rdquo; is the whole story: encoder-only is all bidirectional, decoder-only is all causal, and the encoder–decoder mixes the two by giving each regime its own stack and bonding them with cross-attention. The prefix-LM (Lesson 6) is the fourth point — a single stack with a block mask that is bidirectional over the prefix and causal over the suffix, conditioning implicitly through shared weights instead of cross-attention.\nFigure 1. The three families, laid on one table.\n2. When to reach for each Encoder-only when the task is understanding a fixed input and the output is a label, a span, or a vector — sentiment, classification, named-entity recognition, sentence embeddings, retrieval. No generation, so no decoder, so it is cheap and the bidirectional reading is pure upside.\nDecoder-only when the task is open-ended generation, when you want one uniform stack that scales cleanly, in-context/few-shot learning, and a single model that does many things via prompting. This is the dominant general-purpose LLM design today. Seq2seq tasks are handled by concatenation (input → output), trading the encoder\u0026rsquo;s bidirectional reading for simplicity.\nEncoder–decoder when the task is a genuine transduction with a clearly separate input and output — machine translation, summarization, grammatical error correction, data-to-text, speech-to-text — especially when reading the input well (bidirectionally) is decisive and the output length differs from the input. The compute-once memory and per-layer cross-attention conditioning are worth the extra stack here.\nThe field\u0026rsquo;s trajectory: 2017–2020 was encoder–decoder (MT) and encoder-only (BERT-era understanding); 2020 onward saw decoder-only scale to general-purpose dominance because one uniform causal stack is simpler to scale and prompt. Encoder–decoders persist where their inductive bias pays — translation and summarization systems, and many multimodal models whose decoder cross-attends to encoded image/audio features (the cross-attention of Lesson 4 reused across modalities).\nFigure 2. When to reach for each.\n3. Model zoo Model Family Notable specifics Original Transformer (Vaswani 2017) enc–dec the prototype; sinusoidal positions, post-norm, trained on WMT translation BERT / RoBERTa encoder-only MLM pretraining, [CLS] pooling, no generation GPT-2 / GPT-3 / Llama / Mistral decoder-only causal, next-token CE, seq2seq by concatenation T5 / mT5 / T5 v1.1 / Flan-T5 enc–dec text-to-text framing, span corruption, relative position bias, RMSNorm/pre-norm BART / mBART enc–dec denoising autoencoder (5 noises), close to original Transformer; strong at summarization Pegasus enc–dec gap-sentence pretraining, purpose-built for summarization UniLM / GLM prefix-LM single stack, prefix mask — the Lesson 6 middle ground Whisper enc–dec audio encoder + text decoder; cross-attention bridges modalities Flamingo / many VLMs dec + cross-attn text decoder cross-attends to a frozen vision encoder\u0026rsquo;s features The last two rows make the broader point: once you see cross-attention as \u0026ldquo;queries from one sequence, keys/values from another,\u0026rdquo; the encoder–decoder pattern generalizes far beyond text-to-text — any time one modality or sequence must condition the generation of another, cross-attention is the bond.\nFigure 3. The abstractions, with faces.\n← Course index · Appendix A — Formulas · Appendix C — Acronyms The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/tutorials/encoder-decoder-lessons/appendix-b-architecture-comparison/","summary":"\u003ch1 id=\"appendix-b--architecture-comparison--model-zoo\"\u003eAppendix B — Architecture Comparison \u0026amp; Model Zoo\u003c/h1\u003e\n\u003cp\u003e\u003cem\u003eTechnical course on encoder–decoder transformers — reference appendix.\u003c/em\u003e\u003c/p\u003e\n\u003ch2 id=\"where-this-fits\"\u003eWhere this fits\u003c/h2\u003e\n\u003cp\u003eThis appendix is the side-by-side that Lessons 1 and 6 build toward. The three transformer families are \u003cem\u003enot\u003c/em\u003e three different inventions — they are the same machinery under different attention masks (and different sources for keys/values). This sheet lays them on one table along the axes that actually distinguish them, then names the real models so the abstractions have faces. Read it after Lesson 6, or keep it open from the start as a map.\u003c/p\u003e","title":"Appendix B — Architecture Comparison \u0026 Model Zoo"},{"content":"Appendix B — The Encoder vs. Decoder Cheat Sheet Technical course on encoder LLM mechanics — Appendix to all lessons.\nWhere this fits This appendix collects every contrast drawn across the ten lessons into one place, organized by where in the stack the difference lives. Use it as a lookup table when you need the short answer, with a pointer back to the lesson that derives it. The single root cause to keep in mind: a decoder is causal and generates the next token; an encoder is bidirectional and emits representations. Every row below is a consequence of that one difference (Lesson 1).\nFigure 1. One deleted mask, and everything that follows from it.\nThe master table Aspect Decoder (GPT/Llama-style) Encoder (BERT-style) Lesson Core object $f(x;\\theta)\\to\\Delta^{V-1}$ (next-token distribution) $g(x;\\theta)\\to\\mathbb R^{n\\times d}$ (contextual vectors) 1 Direction causal (left-to-right) bidirectional (both sides) 1, 3 Factorization autoregressive $\\prod_t P(x_t\\mid x_{\\lt t})$ none (no valid joint) 1, 7 Purpose generate text understand / represent text 1 Tokenizer typically BPE typically WordPiece (## marks) 2 Input embedding token + position token + segment + position 2 Special tokens minimal (chat markers) [CLS], [SEP], [MASK], [PAD] 2 Position scheme RoPE / ALiBi (extrapolates) learned-absolute, ≤512 (ModernBERT: RoPE) 2 Attention mask causal + padding padding only (no causal) 3 Attention matrix lower-triangular full $n\\times n$ 3 Block norm trend pre-LN original post-LN 4 Privileged position last token (sees whole prefix) [CLS], position 0 (by convention) 4 Pretraining objective next-token prediction masked language modeling (denoising) 5 Signal density 100% of positions scored ~15% of positions scored 5, 8 Coherence signal implicit in next-token NSP (dropped) / SOP 5 Output head one vocabulary projection, everywhere typed heads per task (pooled / per-token / span) 6 Sequence summary last hidden state (automatic) pooling choice ([CLS] / mean) 6 Intrinsic metric perplexity (well-defined) undefined; pseudo-perplexity stand-in 7 Primary evaluation zero-/few-shot generation quality GLUE/SQuAD/MTEB after fine-tuning 7 Inference shape $n$ sequential passes + KV cache 1 parallel pass, no KV cache 1, 3 Adaptation SFT / RLHF / DPO / prompting attach typed head + supervised FT 9 Needs labeled data to adapt? often no (few-shot) yes 9 Output type sampled text structured label / tags / span / vector 6, 9 Can it generate? yes (its whole point) no 1, 3 Figure 2. The master contrast, selected rows.\nThe one-paragraph summary A decoder hides the future with a causal mask so it can be trained to predict the next token, which makes it a generator: it factorizes sequences autoregressively, runs an $n$-step KV-cached loop at inference, routes every task through one vocabulary head, is measured by perplexity, and is adapted by alignment or prompting. An encoder deletes that mask, so every token sees the whole sequence; this kills next-token training (the answer is visible), forcing the masked-language-modeling denoising objective that scores only ~15% of positions; it emits a matrix of contextual representations rather than a distribution, so it needs typed heads and pooling to make predictions, has no perplexity (no valid joint), runs in a single parallel pass with no KV cache, and is adapted by supervised fine-tuning on labeled data. Decoders generate; encoders understand; encoder–decoders transform; and real systems (RAG) use them together.\nDecision rule, compressed flowchart TD Q{\"Do you need toproduce new text?\"} Q --\u003e|yes| G{\"Transform a longgiven input into output?\"} Q --\u003e|\"no — classify, tag,extract, or embed\"| E[\"ENCODER(single pass, typed head)\"] G --\u003e|yes| ED[\"ENCODER–DECODER(T5 / BART)\"] G --\u003e|\"no — free-formchat / completion\"| D[\"DECODER(autoregressive)\"] E --\u003e RAG{\"Building searchor RAG?\"} RAG --\u003e|yes| BOTH[\"BOTH:encoder retrieves,decoder generates\"] Figure 3. The decision rule, compressed.\n← Course index · Extends every lesson\nAppendices: A — Encoder Family · B — Encoder vs. Decoder Cheat Sheet · C — Acronyms The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/tutorials/encoder-deep-dive/appendix-b-encoder-vs-decoder/","summary":"\u003ch1 id=\"appendix-b--the-encoder-vs-decoder-cheat-sheet\"\u003eAppendix B — The Encoder vs. Decoder Cheat Sheet\u003c/h1\u003e\n\u003cp\u003e\u003cem\u003eTechnical course on encoder LLM mechanics — Appendix to all lessons.\u003c/em\u003e\u003c/p\u003e\n\u003chr\u003e\n\u003ch2 id=\"where-this-fits\"\u003eWhere this fits\u003c/h2\u003e\n\u003cp\u003eThis appendix collects every contrast drawn across the ten lessons into one place, organized by where in the stack the difference lives. Use it as a lookup table when you need the short answer, with a pointer back to the lesson that derives it. The single root cause to keep in mind: \u003cstrong\u003ea decoder is causal and generates the next token; an encoder is bidirectional and emits representations.\u003c/strong\u003e Every row below is a consequence of that one difference (Lesson 1).\u003c/p\u003e","title":"Appendix B — Encoder vs. Decoder"},{"content":"Appendix B — Mathematical Formulas How to read this list. Every load-bearing equation from the ten lessons is gathered here in one place, grouped by topic and ordered roughly as they appear in the course. Each entry gives the formula and a one-line gloss of what it does and where to find its derivation. Math uses LaTeX delimiters ($...$, $$...$$) and renders in any LaTeX-aware Markdown viewer. A symbol glossary closes the appendix.\nNotation at a glance $x_0$ is clean data; $x_1,\\dots,x_T$ are increasingly noisy latents; $x_T\\approx$ pure noise. The schedule is $\\beta_t\\in(0,1)$ with\n$$ \\alpha_t := 1-\\beta_t, \\qquad \\bar\\alpha_t := \\prod_{s=1}^{t}\\alpha_s . $$$q$ is the fixed forward (noising) law, $p_\\theta$ the learned reverse (denoising) law, $\\epsilon\\sim\\mathcal N(0,I)$ standard noise, and $\\epsilon_\\theta$ the network\u0026rsquo;s noise predictor. Throughout, $a_t:=\\sqrt{\\bar\\alpha_t}$ and $b_t:=\\sqrt{1-\\bar\\alpha_t}$.\n1 · The two processes (Lesson 1) Reverse generative joint — the model factorizes as a Markov chain from noise to data:\n$$ p_\\theta(x_{0:T}) = p(x_T)\\prod_{t=1}^{T} p_\\theta(x_{t-1}\\mid x_t), \\qquad p(x_T)=\\mathcal N(0,I). $$Fixed forward chain — the frozen \u0026ldquo;encoder\u0026rdquo; that defines the latents:\n$$ q(x_{1:T}\\mid x_0)=\\prod_{t=1}^T q(x_t\\mid x_{t-1}), \\qquad q(x_t\\mid x_{t-1})=\\mathcal N\\!\\big(\\sqrt{1-\\beta_t}\\,x_{t-1},\\,\\beta_t I\\big). $$Assumed Gaussian reverse step — what the network must parameterize:\n$$ p_\\theta(x_{t-1}\\mid x_t) = \\mathcal N\\!\\left(x_{t-1};\\ \\mu_\\theta(x_t,t),\\ \\Sigma_\\theta(x_t,t)\\right). $$2 · The forward process (Lesson 2) Single-step reparameterization:\n$$ x_t = \\sqrt{1-\\beta_t}\\, x_{t-1} + \\sqrt{\\beta_t}\\,\\epsilon_{t-1}, \\qquad \\epsilon_{t-1}\\sim\\mathcal{N}(0,I). $$Closed-form jump to any noise level — the single most useful identity in diffusion; lets you reach step $t$ in one shot:\n$$ \\boxed{\\,q(x_t\\mid x_0) = \\mathcal{N}\\!\\left(x_t;\\ \\sqrt{\\bar\\alpha_t}\\,x_0,\\ (1-\\bar\\alpha_t) I\\right)\\,} \\quad\\Longleftrightarrow\\quad x_t = \\sqrt{\\bar\\alpha_t}\\,x_0 + \\sqrt{1-\\bar\\alpha_t}\\,\\epsilon,\\ \\ \\epsilon\\sim\\mathcal{N}(0,I). $$Signal-to-noise ratio:\n$$ \\mathrm{SNR}(t) := \\frac{\\bar\\alpha_t}{1-\\bar\\alpha_t}, \\qquad \\log\\mathrm{SNR}(t) = \\log\\bar\\alpha_t - \\log(1-\\bar\\alpha_t). $$Cosine schedule (Nichol \u0026amp; Dhariwal):\n$$ \\bar\\alpha_t = \\frac{f(t)}{f(0)}, \\qquad f(t) = \\cos\\!\\left(\\frac{t/T + s}{1+s}\\cdot\\frac{\\pi}{2}\\right)^2 . $$Continuous-time (VP-SDE) limit: $\\ dx = -\\tfrac12\\beta(t)\\,x\\,dt + \\sqrt{\\beta(t)}\\,dw\\ $ with $\\ \\bar\\alpha_t=\\exp\\!\\big(-\\!\\int_0^t\\beta(s)\\,ds\\big)$.\n3 · The reverse process \u0026amp; tractable posterior (Lesson 3) Bayes form of the posterior — tractable once we condition on $x_0$:\n$$ q(x_{t-1}\\mid x_t, x_0) = q(x_t\\mid x_{t-1})\\,\\frac{q(x_{t-1}\\mid x_0)}{q(x_t\\mid x_0)} = \\mathcal{N}\\!\\left(x_{t-1};\\ \\tilde\\mu_t(x_t, x_0),\\ \\tilde\\beta_t I\\right). $$Posterior variance:\n$$ \\boxed{\\ \\tilde\\beta_t = \\frac{1-\\bar\\alpha_{t-1}}{1-\\bar\\alpha_t}\\,\\beta_t\\ } $$Posterior mean (in $x_0,x_t$ form):\n$$ \\boxed{\\ \\tilde\\mu_t(x_t,x_0) = \\frac{\\sqrt{\\bar\\alpha_{t-1}}\\,\\beta_t}{1-\\bar\\alpha_t}\\,x_0 + \\frac{\\sqrt{\\alpha_t}\\,(1-\\bar\\alpha_{t-1})}{1-\\bar\\alpha_t}\\,x_t\\ } $$Inverting the jump for $x_0$: $\\ x_0 = \\frac{1}{\\sqrt{\\bar\\alpha_t}}\\left(x_t - \\sqrt{1-\\bar\\alpha_t}\\,\\epsilon\\right)$.\nPosterior mean (in noise form) — substitute the line above:\n$$ \\boxed{\\ \\tilde\\mu_t(x_t, x_0) = \\frac{1}{\\sqrt{\\alpha_t}}\\left(x_t - \\frac{\\beta_t}{\\sqrt{1-\\bar\\alpha_t}}\\,\\epsilon\\right)\\ } $$4 · The variational lower bound (Lesson 4) Jensen bound — turns an intractable log-integral into an expectation we can optimize:\n$$ -\\log p_\\theta(x_0) = -\\log \\int q(x_{1:T}\\mid x_0)\\,\\frac{p_\\theta(x_{0:T})}{q(x_{1:T}\\mid x_0)}\\,dx_{1:T} \\le \\mathbb{E}_{q}\\!\\left[-\\log\\frac{p_\\theta(x_{0:T})}{q(x_{1:T}\\mid x_0)}\\right] =: L. $$Exact KL-gap identity — the bound\u0026rsquo;s slack is exactly a KL, hence $L\\ge -\\log p_\\theta(x_0)$:\n$$ \\boxed{\\ L - \\big(-\\log p_\\theta(x_0)\\big) = D_{\\mathrm{KL}}\\!\\big(q(x_{1:T}\\mid x_0)\\,\\|\\,p_\\theta(x_{1:T}\\mid x_0)\\big)\\ \\ge 0.\\ } $$ELBO decomposition into per-step terms:\n$$ \\boxed{, L = \\underbrace{D_{\\mathrm{KL}}!\\big(q(x_T\\mid x_0),|,p(x_T)\\big)}_{L_T}\n\\sum_{t=2}^{T}\\underbrace{D_{\\mathrm{KL}}!\\big(q(x_{t-1}\\mid x_t,x_0),|,p_\\theta(x_{t-1}\\mid x_t)\\big)}{L{t-1}} \\underbrace{-,\\log p_\\theta(x_0\\mid x_1)}_{L_0} ,} $$ Gaussian KL (general, diagonal):\n$$ D_{\\mathrm{KL}} = \\frac{1}{2}\\!\\left[\\operatorname{tr}(\\Sigma_2^{-1}\\Sigma_1) + (\\mu_2-\\mu_1)^\\top\\Sigma_2^{-1}(\\mu_2-\\mu_1) - d + \\log\\frac{\\det\\Sigma_2}{\\det\\Sigma_1}\\right]. $$Denoising term under matched covariance — the KL collapses to a scaled mean gap:\n$$ \\boxed{\\ L_{t-1} = \\frac{1}{2\\sigma_t^2}\\,\\big\\|\\tilde\\mu_t(x_t,x_0) - \\mu_\\theta(x_t,t)\\big\\|^2 + C.\\ } $$5 · The simplified objective (Lesson 5) Mean parameterization through predicted noise:\n$$ \\boxed{\\ \\mu_\\theta(x_t, t) = \\frac{1}{\\sqrt{\\alpha_t}}\\left(x_t - \\frac{\\beta_t}{\\sqrt{1-\\bar\\alpha_t}}\\,\\epsilon_\\theta(x_t,t)\\right)\\ } $$Per-timestep ELBO weight (what $L_{\\text{simple}}$ drops by setting it to $1$):\n$$ \\boxed{\\ w_t = \\frac{\\beta_t^2}{2\\sigma_t^2\\,\\alpha_t(1-\\bar\\alpha_t)}\\ } $$Fully explicit ELBO denoising term:\n$$ \\boxed{\\ L_{t-1} = \\frac{\\beta_t^2}{2\\sigma_t^2\\,\\alpha_t(1-\\bar\\alpha_t)}\\, \\Big\\|\\,\\epsilon - \\epsilon_\\theta\\big(\\sqrt{\\bar\\alpha_t}\\,x_0 + \\sqrt{1-\\bar\\alpha_t}\\,\\epsilon,\\ t\\big)\\,\\Big\\|^2\\ } $$The simplified objective — the famous one-line loss:\n$$ \\boxed{\\ L_{\\text{simple}}(\\theta) = \\mathbb{E}_{t\\sim\\mathcal U\\{1,T\\},\\ x_0\\sim q,\\ \\epsilon\\sim\\mathcal N(0,I)}\\Big[\\big\\|\\,\\epsilon - \\epsilon_\\theta(\\sqrt{\\bar\\alpha_t}\\,x_0 + \\sqrt{1-\\bar\\alpha_t}\\,\\epsilon,\\ t)\\,\\big\\|^2\\Big]\\ } $$v-parameterization — a balanced regression target:\n$$ \\boxed{\\ v := \\sqrt{\\bar\\alpha_t}\\,\\epsilon - \\sqrt{1-\\bar\\alpha_t}\\,x_0\\ } $$with inverse maps $\\ \\epsilon = \\sqrt{1-\\bar\\alpha_t}\\,x_t + \\sqrt{\\bar\\alpha_t}\\,v\\ $ and $\\ x_0 = \\sqrt{\\bar\\alpha_t}\\,x_t - \\sqrt{1-\\bar\\alpha_t}\\,v$.\nScore connection: $\\ \\epsilon_\\theta(x_t,t)\\approx -\\sqrt{1-\\bar\\alpha_t}\\,\\nabla_{x_t}\\log q(x_t)$.\n6 · U-Net \u0026amp; time conditioning (Lesson 6) Sinusoidal time embedding:\n$$ \\text{emb}(t)_{2i} = \\sin\\!\\left(\\frac{t}{10000^{2i/d}}\\right),\\qquad \\text{emb}(t)_{2i+1} = \\cos\\!\\left(\\frac{t}{10000^{2i/d}}\\right),\\qquad i=0,\\dots,\\tfrac{d}{2}-1. $$Time MLP: $\\ \\tau = W_2\\,\\mathrm{SiLU}(W_1\\,\\text{emb}(t) + b_1) + b_2$.\nFiLM / adaLN scale-shift injection:\n$$ \\boxed{\\ (\\gamma,\\beta)=W_\\tau\\,\\tau,\\qquad h \\leftarrow (1+\\gamma)\\odot \\mathrm{Norm}(h) + \\beta\\ } $$adaLN-Zero block (DiT): $\\ h\\leftarrow h+\\alpha\\odot\\mathrm{Block}\\big((1+\\gamma)\\,\\mathrm{LN}(h)+\\beta\\big)$, with $\\alpha$ initialized to $0$.\n7 · The training algorithm (Lesson 7) Algorithm 1 — Training. Repeat: sample $x_0\\sim q(x_0)$, $t\\sim\\mathcal U\\{1,\\dots,T\\}$, $\\epsilon\\sim\\mathcal N(0,I)$; form $x_t = \\sqrt{\\bar\\alpha_t}\\,x_0 + \\sqrt{1-\\bar\\alpha_t}\\,\\epsilon$; take a gradient step on $\\nabla_\\theta \\|\\epsilon - \\epsilon_\\theta(x_t, t)\\|^2$.\nImportance-sampling estimator for the full VLB (variance reduction):\n$$ \\hat L = \\frac{1}{p(t)}\\,L_{t-1},\\qquad \\mathbb{E}_{t\\sim p}[\\hat L] = \\sum_t L_{t-1},\\qquad p(t)\\propto\\sqrt{\\mathbb{E}[L_{t-1}^2]}. $$Min-SNR-$\\gamma$ weight — rebalances easy vs. hard noise levels:\n$$ w_t = \\frac{\\min\\{\\mathrm{SNR}(t),\\,\\gamma\\}}{\\mathrm{SNR}(t)},\\qquad \\mathrm{SNR}(t) = \\frac{\\bar\\alpha_t}{1-\\bar\\alpha_t}. $$8 · DDPM sampling (Lesson 8) Ancestral reverse step — the inference workhorse:\n$$ \\boxed{\\ x_{t-1} = \\frac{1}{\\sqrt{\\alpha_t}}\\left(x_t - \\frac{1-\\alpha_t}{\\sqrt{1-\\bar\\alpha_t}}\\,\\epsilon_\\theta(x_t, t)\\right) + \\sigma_t\\, z, \\qquad z\\sim\\mathcal N(0,I)\\ } $$Two variance choices: $\\ \\sigma_t^2 = \\beta_t\\ $ (upper bound) or $\\ \\sigma_t^2 = \\tilde\\beta_t = \\frac{1-\\bar\\alpha_{t-1}}{1-\\bar\\alpha_t}\\,\\beta_t\\ $ (true posterior).\nPredicted clean image (Tweedie): $\\ \\hat x_0 = \\frac{1}{\\sqrt{\\bar\\alpha_t}}\\big(x_t - \\sqrt{1-\\bar\\alpha_t}\\,\\epsilon_\\theta(x_t,t)\\big)$.\nLearned interpolated variance: $\\ \\sigma_t^2 = \\exp\\!\\big(v\\log\\beta_t + (1-v)\\log\\tilde\\beta_t\\big),\\ v=v_\\theta(x_t,t)\\in[0,1]$.\nAlgorithm 2 — Sampling. Draw $x_T\\sim\\mathcal N(0,I)$; for $t=T,\\dots,1$ set $z\\sim\\mathcal N(0,I)$ if $t\\gt 1$ else $z=0$, apply the boxed update; return $x_0$.\n9 · DDIM \u0026amp; the score view (Lesson 9) Non-Markovian forward family (the DDIM construction):\n$$ q_\\sigma(x_{t-1}\\mid x_t,x_0)=\\mathcal N\\!\\Big(\\sqrt{\\bar\\alpha_{t-1}}\\,x_0+\\sqrt{1-\\bar\\alpha_{t-1}-\\sigma_t^2}\\;\\frac{x_t-\\sqrt{\\bar\\alpha_t}\\,x_0}{\\sqrt{1-\\bar\\alpha_t}},\\;\\sigma_t^2 I\\Big). $$DDIM reverse update:\n$$ \\boxed{\\, x_{t-1} = \\sqrt{\\bar\\alpha_{t-1}}\\;\\underbrace{\\left(\\frac{x_t - \\sqrt{1-\\bar\\alpha_t}\\,\\epsilon_\\theta(x_t,t)}{\\sqrt{\\bar\\alpha_t}}\\right)}_{\\hat x_0 \\text{ (predicted clean image)}} \\;+\\; \\underbrace{\\sqrt{1-\\bar\\alpha_{t-1}-\\sigma_t^2}\\;\\epsilon_\\theta(x_t,t)}_{\\text{direction pointing to } x_t} \\;+\\; \\sigma_t z \\,} $$Deterministic DDIM ($\\sigma_t=0$): $\\ x_{t-1}=\\sqrt{\\bar\\alpha_{t-1}}\\,\\hat x_0+\\sqrt{1-\\bar\\alpha_{t-1}}\\;\\epsilon_\\theta(x_t,t)$. Interpolate with $\\ \\sigma_t=\\eta\\,\\sqrt{\\tilde\\beta_t}$.\nScore ↔ noise identity — the bridge to score-based models:\n$$ \\boxed{\\ \\nabla_{x_t}\\log q_t(x_t) \\approx -\\frac{\\epsilon_\\theta(x_t,t)}{\\sqrt{1-\\bar\\alpha_t}}\\ } $$Tweedie\u0026rsquo;s formula: $\\ \\mathbb E[\\sqrt{\\bar\\alpha_t}\\,x_0\\mid x_t]=x_t+(1-\\bar\\alpha_t)\\,\\nabla_{x_t}\\log q_t(x_t)$.\nAnnealed Langevin dynamics: $\\ x \\leftarrow x + \\tfrac{\\delta}{2}\\,\\nabla_x\\log p(x) + \\sqrt{\\delta}\\,z$.\nAnderson reverse-time SDE: $\\ dx=\\big[f(x,t)-g(t)^2\\,\\nabla_x\\log q_t(x)\\big]dt+g(t)\\,d\\bar w$.\nProbability-flow ODE (deterministic, shares the SDE\u0026rsquo;s marginals):\n$$ \\frac{dx}{dt} = f(x,t) - \\tfrac{1}{2}g(t)^2\\,\\nabla_x\\log q_t(x). $$10 · Conditioning \u0026amp; guidance (Lesson 10) Conditional training loss: $\\ L = \\mathbb{E}_{t,x_0,\\epsilon,c}\\big[\\|\\epsilon - \\epsilon_\\theta(x_t, t, c)\\|^2\\big]$.\nBayes decomposition of the conditional score:\n$$ \\nabla_{x_t}\\log q_t(x_t\\mid c) = \\nabla_{x_t}\\log q_t(x_t) + \\nabla_{x_t}\\log p(c\\mid x_t). $$Classifier guidance:\n$$ \\boxed{\\ \\hat\\epsilon = \\epsilon_\\theta(x_t,t) - w\\,\\sqrt{1-\\bar\\alpha_t}\\;\\nabla_{x_t}\\log p_\\phi(c\\mid x_t)\\ } $$Classifier-free guidance — extrapolate between conditional and unconditional predictions:\n$$ \\boxed{\\ \\hat\\epsilon(x_t,t,c) = \\epsilon_\\theta(x_t,t,\\varnothing) + w\\,\\big(\\epsilon_\\theta(x_t,t,c) - \\epsilon_\\theta(x_t,t,\\varnothing)\\big)\\ } $$Cross-attention conditioning:\n$$ Q=\\phi(x_t)W_Q\\;(N\\times d_k),\\quad K=\\tau(c)W_K\\;(M\\times d_k),\\quad V=\\tau(c)W_V\\;(M\\times d_v), $$ $$ \\text{Attention}=\\mathrm{softmax}\\!\\Big(\\tfrac{QK^\\top}{\\sqrt{d_k}}\\Big)V\\;\\in\\mathbb R^{N\\times d_v}. $$Symbol glossary Symbol Meaning $x_0$ clean data sample $x_t$ noisy latent at step $t$; $x_T\\approx\\mathcal N(0,I)$ $T$ number of diffusion steps $\\beta_t$ forward variance schedule, $\\beta_t\\in(0,1)$ $\\alpha_t$ $1-\\beta_t$ $\\bar\\alpha_t$ $\\prod_{s\\le t}\\alpha_s$ (cumulative signal retention) $a_t,\\,b_t$ shorthand $\\sqrt{\\bar\\alpha_t}$ and $\\sqrt{1-\\bar\\alpha_t}$ $q(\\cdot)$ fixed forward / noising law $p_\\theta(\\cdot)$ learned reverse / denoising law $\\epsilon$ standard Gaussian noise $\\sim\\mathcal N(0,I)$ $\\epsilon_\\theta$ network\u0026rsquo;s predicted noise $\\mu_\\theta,\\Sigma_\\theta$ learned reverse mean and covariance $\\tilde\\mu_t,\\tilde\\beta_t$ true posterior mean and variance $\\sigma_t$ sampling variance at step $t$ $\\hat x_0$ network\u0026rsquo;s predicted clean image $v$ the v-prediction target $w$ guidance / loss-weight scale $c$ conditioning signal (class, text) $\\varnothing$ the null / unconditional token $\\tau$ time (or condition) embedding vector $D_{\\mathrm{KL}}$ Kullback–Leibler divergence $\\nabla_{x}\\log q(x)$ the score function $f,\\,g$ SDE drift and diffusion coefficients ← Appendix A: Acronyms · Course Index ","permalink":"https://www.fanfani.net/tutorials/diffusion-lessons/appendix-b-formulas/","summary":"\u003ch1 id=\"appendix-b--mathematical-formulas\"\u003eAppendix B — Mathematical Formulas\u003c/h1\u003e\n\u003cblockquote\u003e\n\u003cp\u003e\u003cstrong\u003eHow to read this list.\u003c/strong\u003e Every load-bearing equation from the ten lessons is gathered here in one place, grouped by topic and ordered roughly as they appear in the course. Each entry gives the formula and a one-line gloss of what it does and where to find its derivation. Math uses LaTeX delimiters (\u003ccode\u003e$...$\u003c/code\u003e, \u003ccode\u003e$$...$$\u003c/code\u003e) and renders in any LaTeX-aware Markdown viewer. A symbol glossary closes the appendix.\u003c/p\u003e","title":"Appendix B — Formulas"},{"content":"Chapter 12 — The Whole Landscape We have come a long way from \u0026ldquo;a model is just a function.\u0026rdquo; We have opened the box on five different kinds of system and walked the full arc from a pile of random knobs to a deployed, helpful assistant. This closing chapter does something the individual chapters could not: it steps back far enough to see the whole landscape at once — how the pieces relate, what they share, where they differ, and how to think about which one suits a given job. If you read nothing else after the foundations, read this.\nOne toolkit, rearranged The single most important thing to carry away is how little there really is at the bottom, and how much comes from rearranging it. Across the entire family of language models — the storyteller, the reader, the translator, the committee — the underlying parts are the same: text becomes points on a map of meaning; attention lets those points exchange information; the result rises through a tall tower of blocks, kept stable by shortcut connections and normalization; and learning happens by the same downhill walk, scoring wrongness and nudging knobs.\nWhat distinguishes the models is mostly what each word is allowed to look at, and what job the whole thing is pointed at. Let a word see only what came before, and train it to predict the next word, and you have a storyteller that writes. Let every word see in both directions, and train it to fill in blanks, and you have a reader that understands but cannot write. Give reading and writing their own stacks, bonded by cross-attention, and you have a translator that reads one thing and writes another. Replace the per-word processing stage with a bank of experts and a router, and you have a committee that knows enormously while working cheaply. Four famous, distinct-seeming systems, all built from one toolkit with the dials set differently.\nDiffusion stands a little apart, because it generates images by refining a whole noisy canvas rather than producing tokens one at a time — but even it reuses the same deep ideas, learning by minimizing an error and steering itself with the very same cross-attention to follow a text prompt. The unity runs deeper than the surface variety suggests.\nFigure 1. Four shapes, one toolkit, the dials set differently.\nThe cast, in one line each It helps to have the whole cast in view at once.\nA decoder model (GPT, Llama, Claude) is a writer. It generates text left to right, one word at a time, and is the all-purpose workhorse behind chatbots, coding assistants, and open-ended generation. When people say \u0026ldquo;large language model,\u0026rdquo; this is usually what they mean.\nAn encoder model (the BERT family) is a reader. It takes in text bidirectionally and produces a deep understanding of it, but cannot generate. It is the quiet, cheap, high-throughput engine behind search, classification, content moderation, and the retrieval that feeds many larger systems.\nAn encoder–decoder model (T5, BART) is a translator. It bolts a reader to a writer to transform one text into another, and it shines at translation and summarization, where the input must be understood completely before the output is composed.\nA mixture-of-experts model (Switch, Mixtral, DeepSeek) is not a separate task but a separate strategy: a way to build any of the above — usually a writer — so that it holds vast knowledge while only activating a small part of itself per word. It trades cheap memory for the quality of a far larger model, at the price of a more delicate training process and a heavier memory footprint when run.\nA diffusion model (Stable Diffusion, DALL·E) is a sculptor. It generates images, audio, video, and more by starting from pure noise and repeatedly refining the whole thing toward a coherent result, guided by a prompt.\nFigure 2. The cast, in one line each.\nTwo philosophies of generation Among the generators, there is a clean philosophical split worth naming, because it captures a real divide in how machines create.\nThe language models generate sequentially: one piece at a time, in order, each piece committed before the next, the output built up like a sentence written word by word and never revised. This suits language, where order is the essence of meaning and each word genuinely depends on the ones before it.\nThe diffusion models generate all at once, then refine: the entire output is present from the start as noise, and the whole thing is sharpened together over many passes, like a photograph developing. This suits images, where everything must be globally coherent at the same time — the two eyes matching, the light consistent across the frame — which is hard to guarantee if you commit to one piece at a time in some arbitrary order.\nNeither is more advanced than the other. They are matched to the shape of what they make. And the boundaries are blurring — researchers have built image generators that work sequentially and language generators that refine all at once — but the two instincts remain the clearest way to organize the zoo of generative models.\nHow to think about choosing If you ever find yourself deciding which of these to reach for, a few plain questions cut through most of the confusion.\nDo you need to generate free-form text, hold a conversation, or write code? That is a writer\u0026rsquo;s job — a decoder model. If you also have no training data and just want it to follow instructions, lean on its raised, instruction-following form and a good prompt.\nDo you need to classify, tag, score, extract, or search through text you already have, cheaply and at scale? That is a reader\u0026rsquo;s job — an encoder model — and its clean, bounded, trustworthy outputs are a real advantage for high-volume or sensitive work.\nDo you need to turn one whole text into a different one — translate it, summarize it? That is the translator\u0026rsquo;s sweet spot, an encoder–decoder, though in practice a strong writer with a good prompt often does the job too.\nDo you want the quality of a giant model but cannot afford to run a giant model\u0026rsquo;s worth of computation on every word? That is the case for a mixture-of-experts design — provided you can afford the memory to keep all those experts loaded.\nDo you need to create images or other rich media from a description? That is diffusion.\nAnd the most important real-world answer is often more than one of them, cooperating. The dominant pattern in applied AI today has a cheap reader retrieve the relevant documents and a capable writer compose an answer from them — each model doing exactly what its shape was built for. These systems are far more often partners than rivals.\nFigure 3. Five plain questions that cut through most of the confusion.\nThe two ideas that hold it all together We opened the book with three foundational ideas. We close by reaffirming the two that, more than any others, will keep you oriented no matter how the field changes.\nThe first is training writes; using reads. A model is built once, in an expensive act of construction that sets its knobs, and then frozen. When you use it, it is not learning — it is reading. Almost every popular misconception about these systems traces back to forgetting this.\nThe second is learned, chosen, and scratch. Some numbers are the model itself, written by training. Some are settings a human dialed in. Some are mere working values, discarded after each run. Keep these three apart and the fog clears: questions like \u0026ldquo;did it learn that?\u0026rdquo; or \u0026ldquo;where does its behavior come from?\u0026rdquo; become answerable rather than mystifying.\nEverything else in this book — attention, the towers of blocks, the readers and writers and committees and sculptors, the downhill walk of learning, the raising of an assistant, the dials of generation — is built on top of those two ideas. The field will keep producing new names and new records, and the specific architectures will keep shifting. But the shape of the thing, the grammar of how these machines are built and what they do, is what you now understand. New developments will mostly be variations on the themes in these chapters: a different rule for what each word may see, a cleverer way to spend compute or memory, a new game to learn from, a faster way to generate. You have the map. The territory will keep changing, but the map will keep working.\nFor the meaning of any term or acronym used along the way, the Glossary collects them all in plain language. Thank you for reading.\n← Chapter 11 · How a Model Thinks in Real Time · Contents · Appendix A · A Field Guide to Real Models → The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/tutorials/inside-the-machines/chapter-12-the-whole-landscape/","summary":"\u003ch1 id=\"chapter-12--the-whole-landscape\"\u003eChapter 12 — The Whole Landscape\u003c/h1\u003e\n\u003cp\u003eWe have come a long way from \u0026ldquo;a model is just a function.\u0026rdquo; We have opened the box on five different kinds of system and walked the full arc from a pile of random knobs to a deployed, helpful assistant. This closing chapter does something the individual chapters could not: it steps back far enough to see the whole landscape at once — how the pieces relate, what they share, where they differ, and how to think about which one suits a given job. If you read nothing else after the foundations, read this.\u003c/p\u003e","title":"The Whole Landscape"},{"content":"Appendix A — A Field Guide to Real Models The main chapters described the kinds of model in the abstract. This appendix puts faces to the families: the real, named systems you will actually hear about, and the one specific thing each one contributed. The most useful way to read a family tree like this is not as a list of products but as a story of fixes — each model usually exists because someone looked at the previous one, named a single flaw, and corrected it. Read this way, the intimidating roll-call of names becomes a short, sensible history.\nThe reader family: BERT and its descendants These are the \u0026ldquo;reading\u0026rdquo; models from Chapter 5 — they take in a whole passage at once, in both directions, and turn it into understanding rather than new text. The entire family is best understood as a series of repairs to one founding model.\nBERT (2018) — the original reader. BERT was the breakthrough that taught a machine to read a sentence by looking at the whole thing at once, left and right together, instead of word by word in one direction. It learned by playing fill-in-the-blank: hide some words and make the model recover them from context. That single combination — read both ways, learn by guessing hidden words — became the foundation of nearly every understanding task. But BERT had real flaws, and its children would pick them off one at a time: it wasn\u0026rsquo;t trained hard or long enough; one of its two training games was useless; only about one word in seven was actually teaching it anything on each pass; it couldn\u0026rsquo;t handle long documents; and much of its bulk was wasted on bookkeeping.\nRoBERTa — \u0026ldquo;BERT was just undercooked.\u0026rdquo; RoBERTa changed nothing about the design. It simply trained the same model properly: far more text, much longer, with bigger batches, and it threw out the useless second training game (a \u0026ldquo;do these two sentences follow each other?\u0026rdquo; task that taught almost nothing). It also reshuffled which words got hidden each time rather than fixing them once. The startling result — same model, dramatically better — proved the original had been badly undertrained. (This is the episode discussed in Chapter 5: sometimes the cheapest big win is simply more and longer.)\nALBERT — \u0026ldquo;the model is bloated; slim it down.\u0026rdquo; ALBERT noticed BERT carried enormous weight in its vocabulary lookup and in repeating similar machinery layer after layer. It compressed the vocabulary storage and reused the same internal machinery across all its layers instead of keeping a fresh copy at each one, yielding a far smaller model. It also swapped the useless sentence game for a smarter one — telling whether two sentences are in the right order or flipped — which genuinely teaches coherence. The tradeoff: lighter to store, but the heavy reuse can make it slower to learn.\nDistilBERT — \u0026ldquo;make it small enough to actually ship.\u0026rdquo; This one is about deployment, not cleverness. It trained a small \u0026ldquo;student\u0026rdquo; to imitate a full-size BERT \u0026ldquo;teacher,\u0026rdquo; copying not just the right answers but the teacher\u0026rsquo;s whole way of leaning toward them. The payoff: about forty percent smaller and sixty percent faster while keeping roughly ninety-seven percent of the quality. It is the standard move whenever you need a big model\u0026rsquo;s smarts on a phone or a budget.\nELECTRA — \u0026ldquo;stop wasting most of every sentence.\u0026rdquo; ELECTRA\u0026rsquo;s insight was that BERT only learned from the handful of words it hid; the rest of the sentence was just scenery. ELECTRA changed the game: a small helper swaps some words for plausible fakes, and the main model must look at every single word and call it real or fake. Because every word is now a teaching moment rather than one in seven, ELECTRA learns far faster, reaching BERT\u0026rsquo;s quality at roughly a quarter of the cost.\nDeBERTa — \u0026ldquo;the way it tracks position is clumsy.\u0026rdquo; Earlier models mashed together two different things about each word — what it is and where it sits — into one blended signal. DeBERTa kept them separate, letting the model reason about meaning and position independently, and reinjected precise position information right where it matters most. This cleaner handling of \u0026ldquo;what versus where\u0026rdquo; kept it atop the accuracy leaderboards for a long stretch, making it a go-to when you want maximum precision.\nModernBERT (2024) — \u0026ldquo;give the reader everything the chatbot era figured out.\u0026rdquo; While the reader family slept, the text-writing models invented a pile of engineering upgrades. ModernBERT is the catch-up: it imports the modern way of tracking position (so it can read documents of eight thousand words rather than choking at five hundred), faster and steadier internal plumbing, and training that includes computer code. The result is a drop-in replacement for BERT that is faster and handles far longer documents — while keeping the original read-both-ways, fill-in-the-blank soul.\nThe through-line is clean: every member fixes one named flaw of BERT — undertraining (RoBERTa), bloat (ALBERT), shipping size (DistilBERT), wasted words (ELECTRA), clumsy positions (DeBERTa), and short memory plus dated engineering (ModernBERT).\nFigure 1. The reader family, read as a story of fixes.\nThe mixture-of-experts family These are the \u0026ldquo;committee of specialists\u0026rdquo; models from Chapter 7 : rather than running every word through one enormous brain, they keep a large bank of experts and, for each word, wake only the few best suited to it. The recurring villain in this family\u0026rsquo;s history is the router — the traffic cop that decides which experts each word visits — which, left alone, gets lazy and funnels everything to a few favorites. Most of the story is about taming it.\nSparsely-gated mixture-of-experts (2017) — the origin. This is the paper that introduced the whole committee idea, with a router that adds a little randomness to spread the load and the first \u0026ldquo;fairness penalties\u0026rdquo; to stop it playing favorites. It proved you could train thousands of specialists and use only a few at a time, and it named the two problems every successor would wrestle with: lazy favoritism and the need for balancing.\nGShard (2020) — \u0026ldquo;make it work across many machines.\u0026rdquo; A committee this big cannot fit on one chip, so GShard worked out how to spread the experts across many machines and shuttle each word to the right one efficiently. It set the standard plumbing — split the experts across devices, cap how many words any one expert handles, penalize imbalance — that every later model inherited.\nSwitch Transformer (2021) — \u0026ldquo;send each word to just one expert.\u0026rdquo; Everyone had assumed each word needed at least two specialists for safety. Switch boldly cut it to one — the simplest possible setup — and showed that with the right guardrails it works fine and runs much faster. It made the whole approach feel robust rather than fragile.\nGLaM (2021) — \u0026ldquo;prove the savings at giant scale.\u0026rdquo; GLaM was a clean, very large demonstration that a model holding over a trillion parameters\u0026rsquo; worth of knowledge could answer each word using only a tiny fraction of itself — delivering the cost-versus-quality bargain at a scale that turned heads.\nST-MoE (2022) — \u0026ldquo;stop it from blowing up or overfitting.\u0026rdquo; Big committees were notoriously unstable to train and hard to adapt afterward. ST-MoE added a guardrail to keep the router\u0026rsquo;s decisions from spiking out of control, plus a careful recipe for fine-tuning — fixing the two practical headaches that had made large sparse models painful to use.\nMixtral (2023) — \u0026ldquo;the open breakout.\u0026rdquo; Mixtral made the approach mainstream and freely available. With eight experts per layer and two used per word, it matched or beat much larger conventional models while costing a fraction to run, proving to the open community that the committee approach was not just a big-lab trick.\nDeepSeek-MoE / V2 / V3 (2024) — \u0026ldquo;more, slimmer specialists, plus a few always-on ones.\u0026rdquo; DeepSeek refined the recipe two ways. Instead of a few wide experts, it used many slim ones, so each could specialize more sharply — plus a couple of \u0026ldquo;always-on\u0026rdquo; generalists every word visits for the common basics. And it dropped the fairness penalty (which slightly distorted the model\u0026rsquo;s real goal) in favor of a gentle nudge that keeps the router balanced without polluting the actual learning. Its largest version scales to hundreds of experts while running only a small slice per word.\nThese crystallized into two house styles: the Mixtral style (a few wide experts, kept fair with a penalty) is simple and dependable; the DeepSeek style (many slim experts plus always-on generalists, kept fair with a gentle nudge) squeezes out more specialization and efficiency at the cost of more moving parts. Later families such as Qwen\u0026rsquo;s mixture-of-experts models largely follow the slim-plus-generalist pattern, confirming it as a modern default.\nFigure 2. The committee family, and the router it kept taming.\nThe sequence-to-sequence zoo, and how the three body plans map to famous names Step all the way back and there are really just three body plans, all built from the same parts but wired differently. Readers (encoder-only) take in a whole input both directions and produce understanding; they do not write. Writers (decoder-only) read left to right and keep going, predicting the next word over and over; this is the design behind essentially every modern chatbot, and it handles translation-style jobs by simply pasting the input in front and continuing. Reader-writers (encoder–decoder) bolt the two together with the cross-attention bridge from Chapter 6 , and are the natural fit when input and output are clearly separate things of different lengths — translation, summarization, speech-to-text.\nThe rule of thumb: read both directions for understanding; read one direction and continue for generation; do both with a bridge for transduction — turning one whole thing into another.\nA little history explains why one plan came to dominate. From roughly 2017 to 2020, reader-writers ruled translation and readers ruled understanding. But from 2020 on, the pure writer design took over the general-purpose world, because one uniform left-to-right stack is the simplest thing to scale and the easiest to steer with a prompt. Reader-writers survive where their built-in strengths still pay off.\nHere are the famous faces, sorted by body plan:\nThe original Transformer (2017) — the reader-writer prototype, built to translate, that started everything. BERT, RoBERTa — the readers: understanding and search, no writing. GPT-2, GPT-3, Llama, Mistral — the writers: the chatbot and general-purpose lineage everyone knows, doing translation-style tasks just by pasting input in front. T5 (and mT5, Flan-T5) — reader-writers that reframed everything as \u0026ldquo;text in, text out,\u0026rdquo; so one model could translate, summarize, and answer with the same setup. BART (and mBART) — reader-writers trained by deliberately corrupting text and learning to repair it; especially strong at summarization. Pegasus — a reader-writer purpose-built for summarization, trained by hiding whole key sentences and regenerating them. UniLM, GLM — the clever middle ground: a single stack that reads its input both directions but writes its output one direction, getting some reader-writer benefit without a separate second half. Whisper — a reader-writer for audio: one half listens, the other writes the transcript, with the bridge connecting ear to pen. Flamingo and many vision-language models — writers with a bridge to a vision system, letting a text model \u0026ldquo;look at\u0026rdquo; an image and describe it. The closing point is the one worth keeping: that bridge between reader and writer is not merely a text trick. The moment you see it as \u0026ldquo;let the thing being written constantly consult the thing being understood,\u0026rdquo; it generalizes everywhere — text to text, speech to text, image to text. It is the universal bond whenever one stream of information must guide the creation of another.\nFigure 3. Three body plans, and the famous names that fit each.\n← Chapter 12 · The Whole Landscape · Contents · Appendix B · Comparison Cheat Sheets → The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/tutorials/inside-the-machines/appendix-a-field-guide-to-models/","summary":"\u003ch1 id=\"appendix-a--a-field-guide-to-real-models\"\u003eAppendix A — A Field Guide to Real Models\u003c/h1\u003e\n\u003cp\u003eThe main chapters described the \u003cem\u003ekinds\u003c/em\u003e of model in the abstract. This appendix puts faces to the families: the real, named systems you will actually hear about, and the one specific thing each one contributed. The most useful way to read a family tree like this is not as a list of products but as a \u003cem\u003estory of fixes\u003c/em\u003e — each model usually exists because someone looked at the previous one, named a single flaw, and corrected it. Read this way, the intimidating roll-call of names becomes a short, sensible history.\u003c/p\u003e","title":"A Field Guide to Real Models"},{"content":"Appendix B — Mathematical Formulas Every key equation from the course, collected in one place and grouped by lesson. Each formula has a one-line reminder of what it says; follow the lesson link for the full derivation.\nRead top to bottom, the equations retrace the whole arc: the model is defined as a function (Lesson 1), built up layer by layer (Lessons 2–5), scored against a target (Lessons 6), and improved by gradient descent and its refinements (Lessons 7–9) before being run to generate text (Lesson 10). Used as a reference, it is a quick way to recover a formula\u0026rsquo;s meaning and find where it was derived. Each symbol here matches its lesson exactly; nothing is restated in a simpler form.\nEquations render with any LaTeX-aware Markdown viewer (VS Code + Markdown+Math, Obsidian, or GitHub).\n← Course index · Appendix A — Acronyms · Appendix C — Attention Variants Lesson 1 — The Big Picture Open Lesson 1 → The model as a function — maps a token sequence and parameters to a next-token distribution:\n$$ f:\\ \\mathcal V^{\\le n}\\times\\Theta \\to \\Delta^{V-1},\\qquad \\hat{\\mathbf p}=f(x;\\theta). $$Autoregressive factorization — a whole sequence is scored by the chain rule of probability:\n$$ P_\\theta(x_1,\\dots,x_n)=\\prod_{t=1}^{n} P_\\theta(x_t\\mid x_{\\lt t}),\\qquad x_{\\lt t}:=(x_1,\\dots,x_{t-1}). $$Empirical risk minimization — training picks the parameters that minimize expected loss:\n$$ \\theta^\\star=\\arg\\min_{\\theta\\in\\mathbb R^P}\\ \\mathcal L(\\theta),\\qquad \\mathcal L(\\theta)=\\mathbb E_{x\\sim\\mathcal D}\\!\\left[-\\sum_{t} \\log P_\\theta(x_t\\mid x_{\\lt t})\\right]. $$Pretraining compute — the $6PD$ rule of thumb (vs. $\\approx 2P$ FLOPs/token at inference):\n$$ C \\approx 6\\,P\\,D \\ \\text{FLOPs}. $$Lesson 2 — Tokenization \u0026amp; Embeddings Open Lesson 2 → Embedding matrix and lookup — one learned row per token; a lookup is a one-hot matrix product:\n$$ E\\in\\mathbb R^{V\\times d},\\qquad \\mathbf x_i = \\mathbf e_i^\\top E \\in\\mathbb R^{d}. $$Embedding gradient — only the rows of tokens that actually appeared get updated (sparse):\n$$ \\frac{\\partial\\mathcal L}{\\partial E_{j,:}}=\\sum_{\\text{positions }i\\,:\\,x_i=j}\\frac{\\partial\\mathcal L}{\\partial \\mathbf x_i}. $$Sinusoidal positional encoding — fixed sine/cosine features at geometrically spaced frequencies:\n$$ PE_{(p,2k)}=\\sin\\!\\big(p\\,/\\,10000^{2k/d}\\big),\\qquad PE_{(p,2k+1)}=\\cos\\!\\big(p\\,/\\,10000^{2k/d}\\big). $$RoPE — rotate each query/key pair by a position-dependent angle:\n$$ \\begin{pmatrix}q'_{2k}\\\\ q'_{2k+1}\\end{pmatrix} =\\begin{pmatrix}\\cos p\\omega_k \u0026 -\\sin p\\omega_k\\\\ \\sin p\\omega_k \u0026 \\cos p\\omega_k\\end{pmatrix} \\begin{pmatrix}q_{2k}\\\\ q_{2k+1}\\end{pmatrix}. $$Lesson 3 — Self-Attention Open Lesson 3 → Query/key/value projections:\n$$ Q=HW_Q\\in\\mathbb R^{n\\times d_k},\\quad K=HW_K\\in\\mathbb R^{n\\times d_k},\\quad V=HW_V\\in\\mathbb R^{n\\times d_v}. $$Scaled dot-product attention — the core operation (with mask $M$):\n$$ \\mathrm{Attn}(Q,K,V)=\\mathrm{softmax}\\!\\Big(\\tfrac{QK^\\top}{\\sqrt{d_k}}+M\\Big)\\,V. $$Per-token view — output is a softmax-weighted average of value vectors:\n$$ \\mathbf o_i=\\sum_{j=1}^{n}A_{ij}\\,\\mathbf v_j,\\qquad A_{ij}=\\frac{\\exp(s_{ij})}{\\sum_{m}\\exp(s_{im})},\\qquad s_{ij}=\\frac{\\mathbf q_i^\\top\\mathbf k_j}{\\sqrt{d_k}}. $$The $\\sqrt{d_k}$ rationale — raw scores have variance $d_k$, so we divide to keep softmax well-scaled:\n$$ \\mathbf q_i^\\top\\mathbf k_j=\\sum_{l=1}^{d_k} q_{il}k_{jl},\\qquad \\mathbb E[\\cdot]=0,\\qquad \\mathrm{Var}[\\cdot]=d_k. $$Softmax Jacobian — governs gradient flow through attention weights:\n$$ \\frac{\\partial a_i}{\\partial s_j}=a_i(\\delta_{ij}-a_j) \\;\\Longleftrightarrow\\; J=\\mathrm{diag}(\\mathbf a)-\\mathbf a\\mathbf a^\\top. $$Causal mask — blocks attention to future positions:\n$$ M_{ij}=\\begin{cases}0,\u0026 j\\le i\\\\[2pt] -\\infty,\u0026 j\\gt i.\\end{cases} $$Multi-head attention — concatenate independent heads and project:\n$$ \\mathrm{MHA}(H)=\\big[\\,\\mathrm{head}_1\\,\\|\\cdots\\|\\,\\mathrm{head}_h\\,\\big]W_O,\\quad \\mathrm{head}_i=\\mathrm{Attn}(HW_Q^{(i)},HW_K^{(i)},HW_V^{(i)}). $$Lesson 4 — The Transformer Block Open Lesson 4 → Pre-norm residual block — normalize, sublayer, add; twice per block:\n$$ \\tilde H = H + \\mathrm{MHA}\\big(\\mathrm{Norm}(H)\\big),\\quad H' = \\tilde H + \\mathrm{MLP}\\big(\\mathrm{Norm}(\\tilde H)\\big). $$Position-wise MLP — up-project, nonlinearity, down-project:\n$$ \\mathrm{MLP}(\\mathbf x)=W_2\\,\\phi\\!\\big(W_1\\mathbf x+\\mathbf b_1\\big)+\\mathbf b_2,\\qquad W_1\\in\\mathbb R^{d_{ff}\\times d},\\ W_2\\in\\mathbb R^{d\\times d_{ff}}. $$Residual Jacobian — the identity term is why deep stacks still train:\n$$ \\frac{\\partial \\mathbf y}{\\partial \\mathbf x}=I+\\frac{\\partial\\mathcal F}{\\partial\\mathbf x}. $$LayerNorm (forward) — standardize per token, then learned affine:\n$$ \\mu=\\tfrac1d\\textstyle\\sum_k x_k,\\quad \\sigma^2=\\tfrac1d\\sum_k(x_k-\\mu)^2,\\quad \\mathrm{LN}(\\mathbf x)=\\gamma\\odot\\frac{\\mathbf x-\\mu}{\\sqrt{\\sigma^2+\\epsilon}}+\\beta. $$LayerNorm (backward) — gradient projects out the mean and normalized-direction:\n$$ \\frac{\\partial\\mathcal L}{\\partial x_i}=\\frac{1}{\\sqrt{\\sigma^2+\\epsilon}}\\Big(g_i-\\tfrac1d\\textstyle\\sum_k g_k-\\hat x_i\\cdot\\tfrac1d\\sum_k g_k\\hat x_k\\Big). $$RMSNorm — divide by root-mean-square, no mean subtraction:\n$$ \\mathrm{RMS}(\\mathbf x)=\\frac{\\mathbf x}{\\sqrt{\\tfrac1d\\sum_k x_k^2+\\epsilon}}\\odot\\gamma. $$Lesson 5 — Output Head \u0026amp; Objective Open Lesson 5 → Logits and softmax — project hidden state to vocabulary scores, then normalize:\n$$ \\mathbf z_i=\\mathbf h_i^\\top W_U\\in\\mathbb R^V,\\qquad \\hat p_w=\\frac{e^{z_w}}{\\sum_{m=1}^{V}e^{z_m}}. $$Sequence log-likelihood and cross-entropy loss:\n$$ \\log P_\\theta(x_{1:n})=\\sum_{t=1}^{n}\\log \\hat p_{\\,t,\\,x_t},\\qquad \\mathcal L(\\theta)=-\\frac1n\\sum_{t=1}^n\\log \\hat p_{\\,t,\\,x_t}. $$The clean logit-gradient identity — predicted minus target:\n$$ \\frac{\\partial\\mathcal L}{\\partial\\mathbf z}=\\hat{\\mathbf p}-\\mathbf y. $$The full parameter set:\n$$ \\theta=\\Big\\{E,\\ \\{W_Q,W_K,W_V,W_O\\}_{\\ell=1}^N,\\ \\{W_1,W_2,\\mathbf b_1,\\mathbf b_2\\}_{\\ell=1}^N,\\ \\{\\gamma,\\beta\\}_{\\ell},\\ W_U\\Big\\}. $$Lesson 6 — The Loss Function Open Lesson 6 → Cross-entropy and its decomposition — entropy plus KL divergence:\n$$ H(p,\\hat p)=-\\sum_{w=1}^{V}p_w\\log\\hat p_w =\\underbrace{-\\sum_w p_w\\log p_w}_{H(p)}\\;+\\;\\underbrace{\\sum_w p_w\\log\\frac{p_w}{\\hat p_w}}_{D_{\\mathrm{KL}}(p\\,\\|\\,\\hat p)}. $$Hence minimizing loss is minimizing KL to the true distribution:\n$$ \\min_\\theta H(p,\\hat p_\\theta)\\iff \\min_\\theta D_{\\mathrm{KL}}(p\\,\\|\\,\\hat p_\\theta). $$With one-hot targets, the loss collapses to negative log-probability of the seen token:\n$$ H(\\mathbf y,\\hat{\\mathbf p})=-\\log\\hat p_{x_t}. $$Perplexity — the exponential of the loss:\n$$ \\mathrm{PPL}=\\exp\\!\\big(\\mathcal L\\big)=\\exp\\!\\Big(-\\tfrac1n\\textstyle\\sum_t\\log\\hat p_{t,x_t}\\Big). $$Lesson 7 — Backpropagation \u0026amp; Gradient Descent Open Lesson 7 → Gradient vector and the gradient-descent update:\n$$ \\nabla_\\theta\\mathcal L=\\Big(\\tfrac{\\partial\\mathcal L}{\\partial\\theta_1},\\dots,\\tfrac{\\partial\\mathcal L}{\\partial\\theta_P}\\Big),\\qquad \\theta\\leftarrow\\theta-\\eta\\,\\nabla_\\theta\\mathcal L. $$Vector-Jacobian products — the backprop primitive that propagates adjoints layer by layer:\n$$ \\bar{\\mathbf a}^{(\\ell-1)}=\\Big(\\frac{\\partial g_\\ell}{\\partial \\mathbf a^{(\\ell-1)}}\\Big)^{\\!\\top}\\bar{\\mathbf a}^{(\\ell)}, \\qquad \\bar\\theta_\\ell=\\Big(\\frac{\\partial g_\\ell}{\\partial \\theta_\\ell}\\Big)^{\\!\\top}\\bar{\\mathbf a}^{(\\ell)}. $$Linear-layer adjoints — the VJPs for a matrix multiply:\n$$ \\bar W=\\bar{\\mathbf u}\\,\\mathbf x^\\top,\\qquad \\bar{\\mathbf x}=W^\\top\\bar{\\mathbf u}. $$Lesson 8 — Optimizers, Learning Rate \u0026amp; Batches Open Lesson 8 → Momentum — step along an EMA of gradients:\n$$ \\mathbf m_t=\\beta\\,\\mathbf m_{t-1}+(1-\\beta)g_t,\\qquad \\theta\\leftarrow\\theta-\\eta\\,\\mathbf m_t. $$Adam moments and their bias correction:\n$$ \\mathbf m_t=\\beta_1\\mathbf m_{t-1}+(1-\\beta_1)g_t,\\qquad \\mathbf v_t=\\beta_2\\mathbf v_{t-1}+(1-\\beta_2)g_t^{\\odot2}, $$$$ \\hat{\\mathbf m}_t=\\frac{\\mathbf m_t}{1-\\beta_1^{\\,t}},\\qquad \\hat{\\mathbf v}_t=\\frac{\\mathbf v_t}{1-\\beta_2^{\\,t}}. $$Adam update and AdamW (decoupled weight decay):\n$$ \\theta\\leftarrow\\theta-\\eta\\,\\frac{\\hat{\\mathbf m}_t}{\\sqrt{\\hat{\\mathbf v}_t}+\\epsilon}, \\qquad \\theta\\leftarrow\\theta-\\eta\\Big(\\frac{\\hat{\\mathbf m}_t}{\\sqrt{\\hat{\\mathbf v}_t}+\\epsilon}+\\lambda\\theta\\Big). $$Cosine learning-rate schedule:\n$$ \\eta_t=\\eta_{\\min}+\\tfrac12(\\eta_{\\max}-\\eta_{\\min})\\Big(1+\\cos\\tfrac{\\pi(t-T_w)}{T-T_w}\\Big). $$Scaling laws — power-law loss in parameters and data, above an irreducible floor:\n$$ \\mathcal L(N_{\\text{params}},D)\\approx \\mathcal L_\\infty + \\Big(\\tfrac{A}{N_{\\text{params}}}\\Big)^{\\alpha}+\\Big(\\tfrac{B}{D}\\Big)^{\\beta}. $$Lesson 9 — Fine-Tuning, SFT \u0026amp; RLHF Open Lesson 9 → SFT loss with response masking — train only on response tokens:\n$$ \\mathcal L_{\\text{SFT}}(\\theta)=-\\sum_{t}\\,\\mathbb 1[t\\in\\text{response}]\\;\\log P_\\theta\\big(y_t\\mid x,\\,y_{\\lt t}\\big). $$LoRA — a low-rank correction to frozen weights:\n$$ W=W_0+\\Delta W,\\qquad \\Delta W=\\frac{\\alpha}{r}\\,BA,\\quad A\\in\\mathbb R^{r\\times k},\\ B\\in\\mathbb R^{d\\times r}. $$Reward-model loss (Bradley–Terry) over preference pairs:\n$$ \\mathcal L_{\\text{RM}}(\\phi)=-\\,\\mathbb E_{(x,y_w,y_l)}\\Big[\\log\\sigma\\big(r_\\phi(x,y_w)-r_\\phi(x,y_l)\\big)\\Big]. $$KL-anchored RL objective and the PPO clipped surrogate:\n$$ \\max_\\theta\\ \\mathbb E_{x\\sim\\mathcal D,\\,y\\sim\\pi_\\theta(\\cdot\\mid x)}\\Big[\\,r_\\phi(x,y)\\;-\\;\\beta\\,\\log\\frac{\\pi_\\theta(y\\mid x)}{\\pi_{\\text{ref}}(y\\mid x)}\\Big], $$$$ \\mathcal L^{\\text{CLIP}}(\\theta)=\\mathbb E_t\\Big[\\min\\big(\\rho_t\\hat A_t,\\ \\mathrm{clip}(\\rho_t,1-\\epsilon,1+\\epsilon)\\,\\hat A_t\\big)\\Big]. $$DPO loss — optimize preferences directly, no separate reward model:\n$$ \\mathcal L_{\\text{DPO}}(\\theta)=-\\,\\mathbb E_{(x,y_w,y_l)}\\Big[\\log\\sigma\\Big(\\beta\\log\\tfrac{\\pi_\\theta(y_w\\mid x)}{\\pi_{\\text{ref}}(y_w\\mid x)}-\\beta\\log\\tfrac{\\pi_\\theta(y_l\\mid x)}{\\pi_{\\text{ref}}(y_l\\mid x)}\\Big)\\Big]. $$Lesson 10 — Inference in Depth Open Lesson 10 → Generation distribution — same autoregressive product, now conditioned on the prompt:\n$$ P_\\theta(y_{1:m}\\mid x)=\\prod_t P_\\theta(y_t\\mid x,y_{\\lt t}). $$Temperature scaling — rescale logits to sharpen or flatten the distribution:\n$$ \\hat p_w(T)=\\frac{e^{z_w/T}}{\\sum_m e^{z_m/T}}. $$Top-p (nucleus) sampling — smallest token set whose mass reaches $p$:\n$$ S=\\arg\\min\\big\\{|S|:\\textstyle\\sum_{w\\in S}\\hat p_w\\ge p\\big\\}. $$KV-cache memory — linear in sequence length, depth, and width ($\\times 2$ for K and V):\n$$ \\text{bytes}=2\\times n\\times N_{\\text{layers}}\\times d\\times (\\text{bytes/elem}). $$ ← Course index · Appendix A — Acronyms · Appendix C — Attention Variants ","permalink":"https://www.fanfani.net/tutorials/llm-lessons/appendix-b-formulas/","summary":"\u003ch1 id=\"appendix-b--mathematical-formulas\"\u003eAppendix B — Mathematical Formulas\u003c/h1\u003e\n\u003cp\u003e\u003cem\u003eEvery key equation from the course, collected in one place and grouped by lesson. Each formula has a one-line reminder of what it says; follow the lesson link for the full derivation.\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eRead top to bottom, the equations retrace the whole arc: the model is defined as a function (Lesson 1), built up layer by layer (Lessons 2–5), scored against a target (Lessons 6), and improved by gradient descent and its refinements (Lessons 7–9) before being run to generate text (Lesson 10). Used as a reference, it is a quick way to recover a formula\u0026rsquo;s meaning and find where it was derived. Each symbol here matches its lesson exactly; nothing is restated in a simpler form.\u003c/em\u003e\u003c/p\u003e","title":"Appendix B — Formulas"},{"content":"Appendix C — Acronyms \u0026amp; Symbols Technical course on encoder LLM mechanics — reference appendix.\nWhere this fits A lookup table to keep open alongside the lessons rather than read front to back. It expands every acronym and explains every recurring symbol in plain language, with the lesson where each is introduced.\nAcronyms Acronym Expansion Plain meaning Lesson LLM Large Language Model a large neural network over text 1 BERT Bidirectional Encoder Representations from Transformers the original encoder model 1 MLM Masked Language Modeling predict hidden tokens from both sides 5 NSP Next-Sentence Prediction is sentence B the real continuation of A? (dropped) 5 SOP Sentence-Order Prediction are these two sentences in the right order? (ALBERT) 5, A RTD Replaced-Token Detection is each token original or a fake? (ELECTRA) 8, A CLS \u0026ldquo;classification\u0026rdquo; token the position-0 read-out register 2, 4 SEP \u0026ldquo;separator\u0026rdquo; token marks segment boundaries 2 AR Autoregressive the decoder\u0026rsquo;s left-to-right factorization 1, 5 BPE Byte-Pair Encoding the decoder\u0026rsquo;s usual subword tokenizer 2 WordPiece — BERT\u0026rsquo;s likelihood-based subword tokenizer 2 LayerNorm / LN Layer Normalization per-vector normalization in each block 4 post-LN / pre-LN — LayerNorm after vs. inside the residual 4 MLP / FFN Multi-Layer Perceptron / Feed-Forward Network the per-token expansion sub-layer 4 GELU / GeGLU Gaussian Error Linear Unit / Gated GELU block nonlinearities 4, A RoPE Rotary Position Embedding relative positions inside the dot product 2, A KV cache Key/Value cache decoder-only generation cache (encoders lack it) 1, 3 PPL / PPPL Perplexity / Pseudo-Perplexity decoder metric / encoder stand-in 7 GLUE / SuperGLUE General Language Understanding Evaluation the standard understanding benchmark suites 7 SQuAD Stanford Question Answering Dataset extractive-QA benchmark 7 NER / POS Named-Entity Recognition / Part-of-Speech token-labeling tasks 6, 9 NLI Natural Language Inference entailment/contradiction/neutral 7, 9 STS / MTEB Semantic Textual Similarity / Massive Text Embedding Benchmark embedding-quality benchmarks 7 SBERT Sentence-BERT bi-encoder for sentence embeddings 10 RAG Retrieval-Augmented Generation encoder retrieves, decoder generates 10 SFT / RLHF / DPO Supervised Fine-Tuning / Reinforcement Learning from Human Feedback / Direct Preference Optimization decoder alignment methods 9 LoRA Low-Rank Adaptation parameter-efficient fine-tuning 9 T5 / BART Text-to-Text Transfer Transformer / Bidirectional and Auto-Regressive Transformers encoder–decoder models 10 AdamW Adam with decoupled Weight decay the standard optimizer 8 Symbols Symbol Meaning Lesson $g(x;\\theta)$ the encoder function: tokens → contextual vectors 1 $f(x;\\theta)$ the decoder function: tokens → next-token distribution 1 $\\theta$ the learned parameter vector ($P$ of them) 1 $x=(x_1,\\dots,x_n)$ input token sequence 1 $\\tilde x$ the corrupted (masked) sequence 5 $\\mathcal M$ set of masked positions ($\\approx 0.15\\,n$) 5 $H^{(\\ell)}\\in\\mathbb R^{n\\times d}$ the residual stream / hidden states at layer $\\ell$ 2, 4 $H^{(N)}$ top-layer contextual representations (the output) 4 $E_{\\text{tok}},E_{\\text{seg}},E_{\\text{pos}}$ token, segment, position embedding matrices 2 $W_Q,W_K,W_V,W_O$ attention projection matrices 3 $A\\in\\mathbb R^{n\\times n}$ the attention matrix (full for encoders) 3 $M$ attention mask ($0$ everywhere for encoder self-attention; $-\\infty$ for padding/causal) 3 $W_U$ output (unembedding) projection, often tied to $E_{\\text{tok}}$ 6 $\\mathbf c$ pooled [CLS] (or mean) vector 6 $V$ vocabulary size ($30{,}522$ for BERT) 2 $d$ model width ($768$ base, $1024$ large) 2, 4 $N$ number of layers ($12$ base, $24$ large) 4 $n,\\ n_{\\max}$ sequence length; max length ($512$) 2, 3 $\\Delta^{V-1}$ probability simplex over the vocabulary 1 $\\mathcal L_{\\text{MLM}}$ masked-language-modeling loss 5 ← Course index · reference appendix\nAppendices: A — Encoder Family · B — Encoder vs. Decoder Cheat Sheet · C — Acronyms ","permalink":"https://www.fanfani.net/tutorials/encoder-deep-dive/appendix-c-acronyms/","summary":"\u003ch1 id=\"appendix-c--acronyms--symbols\"\u003eAppendix C — Acronyms \u0026amp; Symbols\u003c/h1\u003e\n\u003cp\u003e\u003cem\u003eTechnical course on encoder LLM mechanics — reference appendix.\u003c/em\u003e\u003c/p\u003e\n\u003chr\u003e\n\u003ch2 id=\"where-this-fits\"\u003eWhere this fits\u003c/h2\u003e\n\u003cp\u003eA lookup table to keep open alongside the lessons rather than read front to back. It expands every acronym and explains every recurring symbol in plain language, with the lesson where each is introduced.\u003c/p\u003e\n\u003chr\u003e\n\u003ch2 id=\"acronyms\"\u003eAcronyms\u003c/h2\u003e\n\u003ctable\u003e\n  \u003cthead\u003e\n      \u003ctr\u003e\n          \u003cth\u003eAcronym\u003c/th\u003e\n          \u003cth\u003eExpansion\u003c/th\u003e\n          \u003cth\u003ePlain meaning\u003c/th\u003e\n          \u003cth\u003eLesson\u003c/th\u003e\n      \u003c/tr\u003e\n  \u003c/thead\u003e\n  \u003ctbody\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003eLLM\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003eLarge Language Model\u003c/td\u003e\n          \u003ctd\u003ea large neural network over text\u003c/td\u003e\n          \u003ctd\u003e1\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003eBERT\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003eBidirectional Encoder Representations from Transformers\u003c/td\u003e\n          \u003ctd\u003ethe original encoder model\u003c/td\u003e\n          \u003ctd\u003e1\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003eMLM\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003eMasked Language Modeling\u003c/td\u003e\n          \u003ctd\u003epredict hidden tokens from both sides\u003c/td\u003e\n          \u003ctd\u003e5\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003eNSP\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003eNext-Sentence Prediction\u003c/td\u003e\n          \u003ctd\u003eis sentence B the real continuation of A? (dropped)\u003c/td\u003e\n          \u003ctd\u003e5\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003eSOP\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003eSentence-Order Prediction\u003c/td\u003e\n          \u003ctd\u003eare these two sentences in the right order? (ALBERT)\u003c/td\u003e\n          \u003ctd\u003e5, A\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003eRTD\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003eReplaced-Token Detection\u003c/td\u003e\n          \u003ctd\u003eis each token original or a fake? (ELECTRA)\u003c/td\u003e\n          \u003ctd\u003e8, A\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003eCLS\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003e\u0026ldquo;classification\u0026rdquo; token\u003c/td\u003e\n          \u003ctd\u003ethe position-0 read-out register\u003c/td\u003e\n          \u003ctd\u003e2, 4\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003eSEP\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003e\u0026ldquo;separator\u0026rdquo; token\u003c/td\u003e\n          \u003ctd\u003emarks segment boundaries\u003c/td\u003e\n          \u003ctd\u003e2\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003eAR\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003eAutoregressive\u003c/td\u003e\n          \u003ctd\u003ethe decoder\u0026rsquo;s left-to-right factorization\u003c/td\u003e\n          \u003ctd\u003e1, 5\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003eBPE\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003eByte-Pair Encoding\u003c/td\u003e\n          \u003ctd\u003ethe decoder\u0026rsquo;s usual subword tokenizer\u003c/td\u003e\n          \u003ctd\u003e2\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003eWordPiece\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003e—\u003c/td\u003e\n          \u003ctd\u003eBERT\u0026rsquo;s likelihood-based subword tokenizer\u003c/td\u003e\n          \u003ctd\u003e2\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003eLayerNorm / LN\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003eLayer Normalization\u003c/td\u003e\n          \u003ctd\u003eper-vector normalization in each block\u003c/td\u003e\n          \u003ctd\u003e4\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003epost-LN / pre-LN\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003e—\u003c/td\u003e\n          \u003ctd\u003eLayerNorm after vs. inside the residual\u003c/td\u003e\n          \u003ctd\u003e4\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003eMLP / FFN\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003eMulti-Layer Perceptron / Feed-Forward Network\u003c/td\u003e\n          \u003ctd\u003ethe per-token expansion sub-layer\u003c/td\u003e\n          \u003ctd\u003e4\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003eGELU / GeGLU\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003eGaussian Error Linear Unit / Gated GELU\u003c/td\u003e\n          \u003ctd\u003eblock nonlinearities\u003c/td\u003e\n          \u003ctd\u003e4, A\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003eRoPE\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003eRotary Position Embedding\u003c/td\u003e\n          \u003ctd\u003erelative positions inside the dot product\u003c/td\u003e\n          \u003ctd\u003e2, A\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003eKV cache\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003eKey/Value cache\u003c/td\u003e\n          \u003ctd\u003edecoder-only generation cache (encoders lack it)\u003c/td\u003e\n          \u003ctd\u003e1, 3\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003ePPL / PPPL\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003ePerplexity / Pseudo-Perplexity\u003c/td\u003e\n          \u003ctd\u003edecoder metric / encoder stand-in\u003c/td\u003e\n          \u003ctd\u003e7\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003eGLUE / SuperGLUE\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003eGeneral Language Understanding Evaluation\u003c/td\u003e\n          \u003ctd\u003ethe standard understanding benchmark suites\u003c/td\u003e\n          \u003ctd\u003e7\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003eSQuAD\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003eStanford Question Answering Dataset\u003c/td\u003e\n          \u003ctd\u003eextractive-QA benchmark\u003c/td\u003e\n          \u003ctd\u003e7\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003eNER / POS\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003eNamed-Entity Recognition / Part-of-Speech\u003c/td\u003e\n          \u003ctd\u003etoken-labeling tasks\u003c/td\u003e\n          \u003ctd\u003e6, 9\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003eNLI\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003eNatural Language Inference\u003c/td\u003e\n          \u003ctd\u003eentailment/contradiction/neutral\u003c/td\u003e\n          \u003ctd\u003e7, 9\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003eSTS / MTEB\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003eSemantic Textual Similarity / Massive Text Embedding Benchmark\u003c/td\u003e\n          \u003ctd\u003eembedding-quality benchmarks\u003c/td\u003e\n          \u003ctd\u003e7\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003eSBERT\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003eSentence-BERT\u003c/td\u003e\n          \u003ctd\u003ebi-encoder for sentence embeddings\u003c/td\u003e\n          \u003ctd\u003e10\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003eRAG\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003eRetrieval-Augmented Generation\u003c/td\u003e\n          \u003ctd\u003eencoder retrieves, decoder generates\u003c/td\u003e\n          \u003ctd\u003e10\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003eSFT / RLHF / DPO\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003eSupervised Fine-Tuning / Reinforcement Learning from Human Feedback / Direct Preference Optimization\u003c/td\u003e\n          \u003ctd\u003edecoder alignment methods\u003c/td\u003e\n          \u003ctd\u003e9\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003eLoRA\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003eLow-Rank Adaptation\u003c/td\u003e\n          \u003ctd\u003eparameter-efficient fine-tuning\u003c/td\u003e\n          \u003ctd\u003e9\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003eT5 / BART\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003eText-to-Text Transfer Transformer / Bidirectional and Auto-Regressive Transformers\u003c/td\u003e\n          \u003ctd\u003eencoder–decoder models\u003c/td\u003e\n          \u003ctd\u003e10\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003eAdamW\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003eAdam with decoupled Weight decay\u003c/td\u003e\n          \u003ctd\u003ethe standard optimizer\u003c/td\u003e\n          \u003ctd\u003e8\u003c/td\u003e\n      \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\u003chr\u003e\n\u003ch2 id=\"symbols\"\u003eSymbols\u003c/h2\u003e\n\u003ctable\u003e\n  \u003cthead\u003e\n      \u003ctr\u003e\n          \u003cth\u003eSymbol\u003c/th\u003e\n          \u003cth\u003eMeaning\u003c/th\u003e\n          \u003cth\u003eLesson\u003c/th\u003e\n      \u003c/tr\u003e\n  \u003c/thead\u003e\n  \u003ctbody\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$g(x;\\theta)$\u003c/td\u003e\n          \u003ctd\u003ethe encoder function: tokens → contextual vectors\u003c/td\u003e\n          \u003ctd\u003e1\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$f(x;\\theta)$\u003c/td\u003e\n          \u003ctd\u003ethe decoder function: tokens → next-token distribution\u003c/td\u003e\n          \u003ctd\u003e1\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$\\theta$\u003c/td\u003e\n          \u003ctd\u003ethe learned parameter vector ($P$ of them)\u003c/td\u003e\n          \u003ctd\u003e1\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$x=(x_1,\\dots,x_n)$\u003c/td\u003e\n          \u003ctd\u003einput token sequence\u003c/td\u003e\n          \u003ctd\u003e1\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$\\tilde x$\u003c/td\u003e\n          \u003ctd\u003ethe corrupted (masked) sequence\u003c/td\u003e\n          \u003ctd\u003e5\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$\\mathcal M$\u003c/td\u003e\n          \u003ctd\u003eset of masked positions ($\\approx 0.15\\,n$)\u003c/td\u003e\n          \u003ctd\u003e5\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$H^{(\\ell)}\\in\\mathbb R^{n\\times d}$\u003c/td\u003e\n          \u003ctd\u003ethe residual stream / hidden states at layer $\\ell$\u003c/td\u003e\n          \u003ctd\u003e2, 4\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$H^{(N)}$\u003c/td\u003e\n          \u003ctd\u003etop-layer contextual representations (the output)\u003c/td\u003e\n          \u003ctd\u003e4\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$E_{\\text{tok}},E_{\\text{seg}},E_{\\text{pos}}$\u003c/td\u003e\n          \u003ctd\u003etoken, segment, position embedding matrices\u003c/td\u003e\n          \u003ctd\u003e2\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$W_Q,W_K,W_V,W_O$\u003c/td\u003e\n          \u003ctd\u003eattention projection matrices\u003c/td\u003e\n          \u003ctd\u003e3\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$A\\in\\mathbb R^{n\\times n}$\u003c/td\u003e\n          \u003ctd\u003ethe attention matrix (full for encoders)\u003c/td\u003e\n          \u003ctd\u003e3\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$M$\u003c/td\u003e\n          \u003ctd\u003eattention mask ($0$ everywhere for encoder self-attention; $-\\infty$ for padding/causal)\u003c/td\u003e\n          \u003ctd\u003e3\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$W_U$\u003c/td\u003e\n          \u003ctd\u003eoutput (unembedding) projection, often tied to $E_{\\text{tok}}$\u003c/td\u003e\n          \u003ctd\u003e6\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$\\mathbf c$\u003c/td\u003e\n          \u003ctd\u003epooled \u003ccode\u003e[CLS]\u003c/code\u003e (or mean) vector\u003c/td\u003e\n          \u003ctd\u003e6\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$V$\u003c/td\u003e\n          \u003ctd\u003evocabulary size ($30{,}522$ for BERT)\u003c/td\u003e\n          \u003ctd\u003e2\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$d$\u003c/td\u003e\n          \u003ctd\u003emodel width ($768$ base, $1024$ large)\u003c/td\u003e\n          \u003ctd\u003e2, 4\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$N$\u003c/td\u003e\n          \u003ctd\u003enumber of layers ($12$ base, $24$ large)\u003c/td\u003e\n          \u003ctd\u003e4\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$n,\\ n_{\\max}$\u003c/td\u003e\n          \u003ctd\u003esequence length; max length ($512$)\u003c/td\u003e\n          \u003ctd\u003e2, 3\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$\\Delta^{V-1}$\u003c/td\u003e\n          \u003ctd\u003eprobability simplex over the vocabulary\u003c/td\u003e\n          \u003ctd\u003e1\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$\\mathcal L_{\\text{MLM}}$\u003c/td\u003e\n          \u003ctd\u003emasked-language-modeling loss\u003c/td\u003e\n          \u003ctd\u003e5\u003c/td\u003e\n      \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\u003chr\u003e\n\u003cp\u003e← \u003ca href=\"/tutorials/encoder-deep-dive/\"\u003eCourse index\u003c/a\u003e\n · reference appendix\u003c/p\u003e","title":"Appendix C — Acronyms"},{"content":"Appendix C — Acronyms \u0026amp; Symbols Technical course on encoder–decoder transformers — reference appendix.\nA lookup table for every acronym and symbol used in the course, expanded in plain language. Keep it open alongside the lessons.\nAcronyms Term Expansion / meaning seq2seq Sequence-to-sequence: map an input sequence $x$ to an output sequence $y$ (translation, summarization). enc–dec Encoder–decoder: the two-stack architecture of this course (T5, BART). MLM Masked Language Modeling: BERT\u0026rsquo;s objective — mask ~15% of tokens, predict them in place (bidirectional, no generation). CE Cross-Entropy: the training loss, $-\\sum_t\\log\\hat p_{t,y_t}$. KV cache Key/Value cache: stored keys/values to avoid recomputation at inference. Enc–dec has two — a growing self-attention cache and a fixed cross-attention cache. FFN / MLP Feed-Forward Network: the position-wise two-layer sublayer, $W_2\\phi(W_1x+b_1)+b_2$, $d_{ff}\\approx4d$. MHA Multi-Head Attention: $h$ parallel attention heads, $d_k=d/h$, recombined by $W_O$. BPE / SentencePiece Subword tokenizers; SentencePiece is T5\u0026rsquo;s. Reduce out-of-vocabulary pressure. prefix-LM Prefix Language Model: single stack with a bidirectional prefix + causal suffix mask (UniLM, GLM); the Lesson 6 middle ground. MAP Maximum A Posteriori: the highest-probability output sequence; beam search approximates it. NMT Neural Machine Translation. BLEU Bilingual Evaluation Understudy: precision-oriented n-gram MT metric with a brevity penalty. ROUGE Recall-Oriented Understudy for Gisting Evaluation: recall-oriented n-gram/LCS summarization metric. LCS Longest Common Subsequence (used by ROUGE-L). KD Knowledge Distillation: train a small student to imitate a large teacher. OOV Out-Of-Vocabulary: a token absent from the model\u0026rsquo;s vocabulary; motivated copy mechanisms. LoRA Low-Rank Adaptation: a parameter-efficient fine-tuning method. RMSNorm Root-Mean-Square Normalization: T5\u0026rsquo;s normalization (no mean subtraction). GELU / ReLU / GeGLU Activation functions used in the FFN (BART uses GELU; T5 ReLU, T5 v1.1 GeGLU). \u0026lt;bos\u0026gt; / \u0026lt;eos\u0026gt; / \u0026lt;pad\u0026gt; Begin-of-sequence / end-of-sequence / padding special tokens. \u0026lt;X\u0026gt;,\u0026lt;Y\u0026gt;,\u0026lt;Z\u0026gt; T5 sentinel tokens marking corrupted spans (Lesson 7). Symbols Symbol Meaning $x=(x_1,\\dots,x_n)$ Input (source) token sequence, length $n$. $y=(y_1,\\dots,y_m)$ Output (target) token sequence, length $m$. $\\tilde x$ Corrupted input used in denoising pretraining (Lesson 7). $\\theta\\in\\mathbb R^P$ All learned parameters; $P$ the parameter count. $d$ Model (hidden) width. $d_k,d_v$ Key/query and value dimensions per head; usually $d_k=d_v=d/h$. $h$ Number of attention heads. $N_{\\text{enc}},N_{\\text{dec}}$ Number of encoder / decoder blocks (depths). $d_{ff}$ FFN inner width, $\\approx 4d$. $V,V_x,V_y$ Vocabulary size (shared / source / target). $E,W_U$ Embedding matrix; unembedding (often $W_U=E^\\top$). $H_{\\text{enc}}\\in\\mathbb R^{n\\times d}$ Encoder output memory — one contextual vector per input token. [STATE]. $H_{\\text{dec}}\\in\\mathbb R^{m\\times d}$ Decoder hidden states. $W_Q,W_K,W_V,W_O$ Attention projection matrices. Superscript $s$ = decoder self-attention; $c$ = cross-attention. $Q,K,V$ Query / key / value matrices (queries from decoder, keys/values from encoder, in cross-attention). $M,M^{\\text{causal}},M^{\\text{pad}},M^{\\text{prefix}}$ Additive attention masks (0 or $-\\infty$): causal, padding, prefix-LM. $A$ Attention matrix; square ($n\\times n$ or $m\\times m$) for self-attention, rectangular ($m\\times n$) for cross-attention. $s_{ij}$ Attention score (scaled dot product) between query $i$ and key $j$. $\\mathbf z_t,\\hat{\\mathbf p}_t$ Logits and predicted probability distribution at output position $t$. $\\mathbf y_t$ One-hot target at position $t$. $\\Delta^{V-1}$ Probability simplex over $V$ tokens (valid categorical distributions). $\\mathcal L$ Loss (cross-entropy unless noted). $T$ Softmax temperature (inference, and distillation softening). $k,\\alpha$ Beam width; length-penalty exponent (Lesson 9). $\\mathbf c^t$ Coverage vector — cumulative attention to each source position (Lesson 9). $p_{\\text{gen}}$ Pointer-generator gate: probability of generating vs. copying (Lesson 9). $\\lambda$ Distillation mixing weight (Lesson 10). $\\sigma,\\phi$ Sigmoid; FFN activation. ← Course index · Appendix A — Formulas · Appendix B — Architecture Comparison ","permalink":"https://www.fanfani.net/tutorials/encoder-decoder-lessons/appendix-c-acronyms/","summary":"\u003ch1 id=\"appendix-c--acronyms--symbols\"\u003eAppendix C — Acronyms \u0026amp; Symbols\u003c/h1\u003e\n\u003cp\u003e\u003cem\u003eTechnical course on encoder–decoder transformers — reference appendix.\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003eA lookup table for every acronym and symbol used in the course, expanded in plain language. Keep it open alongside the lessons.\u003c/p\u003e\n\u003chr\u003e\n\u003ch2 id=\"acronyms\"\u003eAcronyms\u003c/h2\u003e\n\u003ctable\u003e\n  \u003cthead\u003e\n      \u003ctr\u003e\n          \u003cth\u003eTerm\u003c/th\u003e\n          \u003cth\u003eExpansion / meaning\u003c/th\u003e\n      \u003c/tr\u003e\n  \u003c/thead\u003e\n  \u003ctbody\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003eseq2seq\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003eSequence-to-sequence: map an input sequence $x$ to an output sequence $y$ (translation, summarization).\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003eenc–dec\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003eEncoder–decoder: the two-stack architecture of this course (T5, BART).\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003eMLM\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003eMasked Language Modeling: BERT\u0026rsquo;s objective — mask ~15% of tokens, predict them in place (bidirectional, no generation).\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003eCE\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003eCross-Entropy: the training loss, $-\\sum_t\\log\\hat p_{t,y_t}$.\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003eKV cache\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003eKey/Value cache: stored keys/values to avoid recomputation at inference. Enc–dec has two — a growing self-attention cache and a fixed cross-attention cache.\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003eFFN / MLP\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003eFeed-Forward Network: the position-wise two-layer sublayer, $W_2\\phi(W_1x+b_1)+b_2$, $d_{ff}\\approx4d$.\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003eMHA\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003eMulti-Head Attention: $h$ parallel attention heads, $d_k=d/h$, recombined by $W_O$.\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003eBPE / SentencePiece\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003eSubword tokenizers; SentencePiece is T5\u0026rsquo;s. Reduce out-of-vocabulary pressure.\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003eprefix-LM\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003ePrefix Language Model: single stack with a bidirectional prefix + causal suffix mask (UniLM, GLM); the Lesson 6 middle ground.\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003eMAP\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003eMaximum A Posteriori: the highest-probability output sequence; beam search approximates it.\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003eNMT\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003eNeural Machine Translation.\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003eBLEU\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003eBilingual Evaluation Understudy: precision-oriented n-gram MT metric with a brevity penalty.\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003eROUGE\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003eRecall-Oriented Understudy for Gisting Evaluation: recall-oriented n-gram/LCS summarization metric.\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003eLCS\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003eLongest Common Subsequence (used by ROUGE-L).\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003eKD\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003eKnowledge Distillation: train a small student to imitate a large teacher.\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003eOOV\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003eOut-Of-Vocabulary: a token absent from the model\u0026rsquo;s vocabulary; motivated copy mechanisms.\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003eLoRA\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003eLow-Rank Adaptation: a parameter-efficient fine-tuning method.\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003eRMSNorm\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003eRoot-Mean-Square Normalization: T5\u0026rsquo;s normalization (no mean subtraction).\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003eGELU / ReLU / GeGLU\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003eActivation functions used in the FFN (BART uses GELU; T5 ReLU, T5 v1.1 GeGLU).\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003e\u003ccode\u003e\u0026lt;bos\u0026gt;\u003c/code\u003e / \u003ccode\u003e\u0026lt;eos\u0026gt;\u003c/code\u003e / \u003ccode\u003e\u0026lt;pad\u0026gt;\u003c/code\u003e\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003eBegin-of-sequence / end-of-sequence / padding special tokens.\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003e\u003ccode\u003e\u0026lt;X\u0026gt;,\u0026lt;Y\u0026gt;,\u0026lt;Z\u0026gt;\u003c/code\u003e\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003eT5 \u003cstrong\u003esentinel\u003c/strong\u003e tokens marking corrupted spans (Lesson 7).\u003c/td\u003e\n      \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\u003chr\u003e\n\u003ch2 id=\"symbols\"\u003eSymbols\u003c/h2\u003e\n\u003ctable\u003e\n  \u003cthead\u003e\n      \u003ctr\u003e\n          \u003cth\u003eSymbol\u003c/th\u003e\n          \u003cth\u003eMeaning\u003c/th\u003e\n      \u003c/tr\u003e\n  \u003c/thead\u003e\n  \u003ctbody\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$x=(x_1,\\dots,x_n)$\u003c/td\u003e\n          \u003ctd\u003eInput (source) token sequence, length $n$.\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$y=(y_1,\\dots,y_m)$\u003c/td\u003e\n          \u003ctd\u003eOutput (target) token sequence, length $m$.\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$\\tilde x$\u003c/td\u003e\n          \u003ctd\u003eCorrupted input used in denoising pretraining (Lesson 7).\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$\\theta\\in\\mathbb R^P$\u003c/td\u003e\n          \u003ctd\u003eAll learned parameters; $P$ the parameter count.\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$d$\u003c/td\u003e\n          \u003ctd\u003eModel (hidden) width.\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$d_k,d_v$\u003c/td\u003e\n          \u003ctd\u003eKey/query and value dimensions per head; usually $d_k=d_v=d/h$.\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$h$\u003c/td\u003e\n          \u003ctd\u003eNumber of attention heads.\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$N_{\\text{enc}},N_{\\text{dec}}$\u003c/td\u003e\n          \u003ctd\u003eNumber of encoder / decoder blocks (depths).\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$d_{ff}$\u003c/td\u003e\n          \u003ctd\u003eFFN inner width, $\\approx 4d$.\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$V,V_x,V_y$\u003c/td\u003e\n          \u003ctd\u003eVocabulary size (shared / source / target).\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$E,W_U$\u003c/td\u003e\n          \u003ctd\u003eEmbedding matrix; unembedding (often $W_U=E^\\top$).\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$H_{\\text{enc}}\\in\\mathbb R^{n\\times d}$\u003c/td\u003e\n          \u003ctd\u003eEncoder output \u003cstrong\u003ememory\u003c/strong\u003e — one contextual vector per input token. \u003cstrong\u003e[STATE]\u003c/strong\u003e.\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$H_{\\text{dec}}\\in\\mathbb R^{m\\times d}$\u003c/td\u003e\n          \u003ctd\u003eDecoder hidden states.\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$W_Q,W_K,W_V,W_O$\u003c/td\u003e\n          \u003ctd\u003eAttention projection matrices. Superscript $s$ = decoder self-attention; $c$ = cross-attention.\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$Q,K,V$\u003c/td\u003e\n          \u003ctd\u003eQuery / key / value matrices (queries from decoder, keys/values from encoder, in cross-attention).\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$M,M^{\\text{causal}},M^{\\text{pad}},M^{\\text{prefix}}$\u003c/td\u003e\n          \u003ctd\u003eAdditive attention masks (0 or $-\\infty$): causal, padding, prefix-LM.\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$A$\u003c/td\u003e\n          \u003ctd\u003eAttention matrix; square ($n\\times n$ or $m\\times m$) for self-attention, rectangular ($m\\times n$) for cross-attention.\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$s_{ij}$\u003c/td\u003e\n          \u003ctd\u003eAttention score (scaled dot product) between query $i$ and key $j$.\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$\\mathbf z_t,\\hat{\\mathbf p}_t$\u003c/td\u003e\n          \u003ctd\u003eLogits and predicted probability distribution at output position $t$.\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$\\mathbf y_t$\u003c/td\u003e\n          \u003ctd\u003eOne-hot target at position $t$.\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$\\Delta^{V-1}$\u003c/td\u003e\n          \u003ctd\u003eProbability simplex over $V$ tokens (valid categorical distributions).\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$\\mathcal L$\u003c/td\u003e\n          \u003ctd\u003eLoss (cross-entropy unless noted).\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$T$\u003c/td\u003e\n          \u003ctd\u003eSoftmax temperature (inference, and distillation softening).\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$k,\\alpha$\u003c/td\u003e\n          \u003ctd\u003eBeam width; length-penalty exponent (Lesson 9).\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$\\mathbf c^t$\u003c/td\u003e\n          \u003ctd\u003eCoverage vector — cumulative attention to each source position (Lesson 9).\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$p_{\\text{gen}}$\u003c/td\u003e\n          \u003ctd\u003ePointer-generator gate: probability of generating vs. copying (Lesson 9).\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$\\lambda$\u003c/td\u003e\n          \u003ctd\u003eDistillation mixing weight (Lesson 10).\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$\\sigma,\\phi$\u003c/td\u003e\n          \u003ctd\u003eSigmoid; FFN activation.\u003c/td\u003e\n      \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\u003chr\u003e\n\u003cp\u003e\u003ca href=\"/tutorials/encoder-decoder-lessons/\"\u003e← Course index\u003c/a\u003e\n · \u003ca href=\"/tutorials/encoder-decoder-lessons/appendix-a-formulas/\"\u003eAppendix A — Formulas\u003c/a\u003e\n · \u003ca href=\"/tutorials/encoder-decoder-lessons/appendix-b-architecture-comparison/\"\u003eAppendix B — Architecture Comparison\u003c/a\u003e\n\u003c/p\u003e","title":"Appendix C — Acronyms \u0026 Symbols"},{"content":"Appendix C — A Field Guide to Attention Variants Technical course on LLM mechanics — Appendix to Lessons 3 and 10.\nWhere this fits This is a side-quest off Lessons 3 and 10. Lesson 3 built the attention mechanism; Lesson 10 showed why, at inference time, the obvious version is too expensive to serve. The names that get thrown around — MQA, GQA, sliding-window, RoPE, FlashAttention — are all answers to one of those two pressures (what the task needs, or what the hardware can afford), and this appendix sorts them out. Lesson 3 defined the one operation everything here is a variant of — scaled dot-product attention,\n$$ \\mathrm{Attn}(Q,K,V)=\\mathrm{softmax}\\!\\Big(\\tfrac{QK^\\top}{\\sqrt{d_k}}+M\\Big)V, $$and wrapped it in multi-head form. Lesson 10 then introduced MQA/GQA and the KV cache as the levers that make generation affordable. This appendix collects every attention variant the field actually uses, organized along the four independent axes on which they differ, and — the point of the appendix — says plainly where each is used and where it is not.\nThe axes are independent: a real model picks one option from each. A modern decoder-only LLM is typically causal (axis 1) + grouped-query (axis 2) + sliding-window or dense (axis 3) + RoPE-relative (axis 4), run through a FlashAttention kernel (an implementation detail, not a fifth axis). Keeping the axes separate is what stops \u0026ldquo;attention variants\u0026rdquo; from feeling like an undifferentiated zoo.\nA reading note on categories (Lesson 1\u0026rsquo;s framing): almost every choice in this appendix is [CONFIG] — architecture decided before training. The projection matrices inside any variant are [LEARNED]; the attention matrix and KV cache are [STATE]. When a variant \u0026ldquo;saves memory,\u0026rdquo; it is almost always shrinking [STATE] (the cache) or recomputed activations, not the [LEARNED] parameter count.\nWe take the axes in order of how much freedom they leave. The first is forced by the task; the rest are increasingly about cost.\nAxis 1 — Connectivity: who is allowed to attend to whom This axis is set by the mask $M$ and by whether queries and keys come from the same sequence. It is the most consequential choice because it follows from the task, not from efficiency.\nBidirectional (full) self-attention. No mask: every position attends to every other, past and future. This is the encoder pattern of the original Transformer and of BERT-style models. Used in: masked-language-model encoders (BERT, RoBERTa), the encoder half of T5, sentence-embedding and retrieval/classification models, and the vision-transformer image encoders that feed multimodal LLMs. Not used in: autoregressive text generators — letting position $t$ see the future trivially defeats next-token prediction, so no GPT-style model uses it for the language head.\nCausal (masked) self-attention. The $-\\infty$ upper-triangular mask from Lesson 3 §4: position $t$ sees only $\\le t$. This is the attention of every decoder-only LLM (GPT, Llama, Mistral, Qwen, Gemma, Claude-class models). It is non-negotiable for generation because it is what makes the autoregressive factorization (Lesson 5) valid and lets all positions train in parallel from a single forward pass. Not used where the whole input is known up front and bidirectional context helps, e.g. an encoder doing classification.\nCross-attention. Queries come from one sequence, keys/values from another: $Q=Y W_Q$, $K,V = X W_{K},X W_V$. This is how a decoder conditions on an encoder\u0026rsquo;s output. Used in: encoder–decoder architectures (the original Transformer, T5, BART) for translation and summarization; multimodal models, where text-token queries attend to image/audio features (Flamingo\u0026rsquo;s gated cross-attention, many vision-language adapters); retrieval-augmented setups that attend over encoded documents. Not used in: pure decoder-only LLMs, which fold \u0026ldquo;conditioning\u0026rdquo; into the prompt and use only self-attention — the dominant design today, which is why cross-attention is now more a multimodal/seq2seq tool than a core LLM one.\nPrefix / non-causal-prefix (bidirectional prompt, causal completion). A hybrid mask: tokens in the prompt attend bidirectionally, generated tokens attend causally (UniLM, T5\u0026rsquo;s \u0026ldquo;prefix-LM\u0026rdquo;). Used in: a minority of unified models and some infilling setups. Not used much in mainstream decoder-only LLMs, which keep a single causal mask for simplicity and KV-cache friendliness.\nAxis 2 — Head structure: how queries, keys, and values are shared All of these are exact attention — same math, same outputs up to the sharing constraint. The axis trades [LEARNED] key/value capacity and, more importantly, [STATE] KV-cache size against quality. This is the axis Lesson 10 §4 cares about, because the KV cache is the binding constraint on long-context inference.\nSingle-head attention. One set of $W_Q,W_K,W_V$. Used in: pedagogy and a few tiny models. Not used in any serious LLM — one head forces a single similarity structure, which Lesson 3 §5 showed is too weak.\nMulti-head attention (MHA). $h$ independent heads, each with full $W_Q^{(i)},W_K^{(i)},W_V^{(i)}$, recombined by $W_O$; $\\approx 4d^2$ params/layer. The default for a decade and still standard at small/medium scale and in most encoders. Used in: the original Transformer, BERT, GPT-2/GPT-3, and most models through ~2022. Increasingly not used at large scale for decoding, because it stores $h$ separate K and V vectors per token — the heaviest possible KV cache.\nMulti-query attention (MQA). $h$ query heads but one shared K head and one shared V head. Shrinks the KV cache (and the decode-time memory-bandwidth bill) by roughly $h\\times$. Used in: PaLM, Falcon, early latency-sensitive deployments. Falling out of favor as the default because the aggressive sharing can cost quality and training stability — it is now usually superseded by its middle ground, GQA.\nGrouped-query attention (GQA). Query heads are partitioned into $g$ groups; each group shares one K/V head ($g=1$ is MQA, $g=h$ is MHA). Captures most of MQA\u0026rsquo;s cache savings with most of MHA\u0026rsquo;s quality. Used in: the current mainstream — Llama 2 70B and Llama 3, Mistral, Mixtral, Qwen2, Gemma 2, and most new large decoders. If you don\u0026rsquo;t know what a modern open LLM uses, GQA is the safe guess. Not used where the model is small enough that the full MHA cache is already cheap, or where MLA (below) has been adopted instead.\nMulti-head latent attention (MLA). Keys and values are compressed into a low-rank latent vector that is cached and decompressed on the fly, so the cache stores the small latent rather than full K/V. Pushes cache savings further than GQA while aiming to preserve MHA-like quality. Used in: DeepSeek-V2/V3 and models following that line. Not yet broadly adopted elsewhere — it is the newest entry on this axis and adds implementation complexity, so most teams still default to GQA.\nFigure 1. Sharing keys and values: four points on one axis.\nAxis 3 — Sparsity / span: how much of the sequence each query sees Full attention is $O(n^2)$ in compute and, naively, in the score memory (Lesson 3 §6). This axis cuts that cost by having each query attend to a subset of positions. Everything here is an approximation of full attention (FlashAttention in Axis 4 is the exception — it is exact). The recurring lesson of this axis: most efficient-attention research is not used in production frontier LLMs, which mostly stayed dense and instead optimized the exact computation.\nDense / full attention. Every query attends to every allowed key. Used in: the vast majority of deployed LLMs at their trained context length — once FlashAttention made exact attention IO-efficient, the pressure to approximate dropped sharply. Not ideal only at very long contexts, where $n^2$ finally bites.\nSliding-window (local) attention. Each query attends to a fixed window of the most recent $w$ tokens; stacking $L$ layers gives an effective receptive field of $\\approx L\\cdot w$. Cost is $O(n\\,w)$, linear in length. Used in: Mistral 7B and Mixtral (window 4096), Longformer\u0026rsquo;s local component, Gemma 2 (alternating local/global layers), and many long-context designs. Not used as the only mechanism when genuinely global, long-range dependencies matter every layer — hence the common pattern of interleaving local layers with occasional full-attention layers.\nDilated / strided sparse attention. Fixed strided or block patterns so a query reaches far-away tokens through gaps (Sparse Transformer, LongNet\u0026rsquo;s dilation). Used in: a handful of long-sequence and long-document research models. Not used much in mainstream chat LLMs — the patterns are hand-designed and brittle relative to dense + good kernels.\nGlobal + local / block-sparse (Longformer, BigBird, ETC). Most tokens are local; a few designated \u0026ldquo;global\u0026rdquo; tokens (e.g. [CLS], question tokens) attend to and are attended by everyone. Provably retains much of full attention\u0026rsquo;s expressivity at linear cost. Used in: long-document encoders for QA and classification. Largely not used in decoder-only generators, where the KV cache and causal masking made these patterns awkward and FlashAttention removed the motivation.\nLinear / kernelized attention (Performer, Linear Transformers, RFA). Replace $\\mathrm{softmax}(QK^\\top)V$ with a kernel feature map $\\phi$ so the product associates as $\\phi(Q)\\big(\\phi(K)^\\top V\\big)$, giving $O(n)$ cost and a recurrent (RNN-like) inference form with $O(1)$ state per step. Used in: efficiency research and some specialized long-context or streaming systems. Not used in frontier LLMs — the softmax approximation has historically cost accuracy. This idea has been partly reborn outside attention as state-space and linear-recurrent models (Mamba, RWKV, RetNet), which compete with attention rather than implement it.\nLow-rank attention (Linformer). Project the length-$n$ K/V down to a fixed $k\\ll n$ before attending, $O(nk)$. Used in: efficiency benchmarks. Not used in production decoders — the projection assumes a fixed/known length and breaks the incremental KV cache that autoregressive decoding depends on.\nFigure 2. Four sparsity patterns, and why production mostly stayed dense.\nAxis 4 — Position handling and exact kernels Two things often filed under \u0026ldquo;attention variants\u0026rdquo; are really modifiers of the same operation: how position enters the scores, and how the kernel is computed. They compose with any choice on Axes 1–3.\nHow position enters the attention score. Lesson 2 introduced positions; the modern choices live inside attention. Absolute (sinusoidal/learned) position is added once at the input — original Transformer, GPT-2, BERT; largely abandoned for new decoders because it extrapolates poorly past the trained length. Relative position bias adds a learned term depending on $i-j$ to the scores — T5, and ALiBi as a fixed linear distance penalty (used in BLOOM, MPT) that extrapolates to longer contexts. RoPE (rotary) rotates Q and K by a position-dependent angle so the dot product depends on relative offset — the dominant choice today (Llama, Mistral, Qwen, Gemma, most new models), and the substrate for long-context tricks like position interpolation, NTK-aware scaling, and YaRN. Not used: plain absolute encodings in any model that targets long or extrapolated context.\nFlashAttention (and FlashAttention-2/3). Not a different attention — exact scaled dot-product attention computed with an IO-aware, tiled, online-softmax kernel that never materializes the full $n\\times n$ score matrix, cutting attention memory from $O(n^2)$ to $O(n)$ and running much faster on GPUs. Used in: essentially all current training and inference stacks; it is the reason the field could keep dense attention instead of fleeing to the Axis-3 approximations. Not used only on hardware/frameworks where the kernel isn\u0026rsquo;t available, which then fall back to the naive $O(n^2)$-memory implementation.\nPagedAttention. A serving-system technique (vLLM): the KV cache is stored in non-contiguous \u0026ldquo;pages\u0026rdquo; like virtual memory, eliminating fragmentation and enabling prefix sharing across requests. Used in: high-throughput inference servers. It manages [STATE], not math — included here only because it is frequently confused with an attention variant. Not relevant to single-request or training settings.\nHow a real model is assembled Axis The choice What modern decoder-only LLMs pick 1 — Connectivity mask / source of K,V Causal self-attention (cross-attention only for multimodal/seq2seq) 2 — Head structure K/V sharing across heads GQA at scale (MHA small models; MLA newest; MQA waning) 3 — Sparsity / span how many keys per query Dense, or sliding-window for long context 4a — Position how offset enters scores RoPE (ALiBi/relative as alternatives) 4b — Kernel how it\u0026rsquo;s computed FlashAttention (exact), PagedAttention for serving The throughline matches the course\u0026rsquo;s invariants. Axis 1 is fixed by whether the task is generative (it is, for an LLM → causal). Axes 2 and 4b are dominated by the [STATE] budget — the KV cache and the score matrix are what blow up at inference, so the winning variants (GQA, MLA, FlashAttention) all attack transient state, not learned parameters. Axis 3\u0026rsquo;s approximations looked necessary in 2020 and mostly lost to exact attention + better kernels, surviving chiefly as sliding windows and as the conceptual seed of the non-attention sequence models now challenging the Transformer. Everything else is the same $\\mathrm{softmax}(QK^\\top/\\sqrt{d_k})V$ from Lesson 3, masked, shared, and computed in whichever way the deployment can afford.\nFigure 3. One choice per axis, and what a modern decoder picks.\n← Course index · Appendix B — Formulas · Related: Lesson 3 — Self-Attention , Lesson 10 — Inference in Depth The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/tutorials/llm-lessons/appendix-c-attention-variants/","summary":"\u003ch1 id=\"appendix-c--a-field-guide-to-attention-variants\"\u003eAppendix C — A Field Guide to Attention Variants\u003c/h1\u003e\n\u003cp\u003e\u003cem\u003eTechnical course on LLM mechanics — Appendix to Lessons 3 and 10.\u003c/em\u003e\u003c/p\u003e\n\u003chr\u003e\n\u003ch2 id=\"where-this-fits\"\u003eWhere this fits\u003c/h2\u003e\n\u003cp\u003eThis is a side-quest off Lessons 3 and 10. Lesson 3 built the attention mechanism; Lesson 10 showed why, at inference time, the obvious version is too expensive to serve. The names that get thrown around — MQA, GQA, sliding-window, RoPE, FlashAttention — are all answers to one of those two pressures (what the task needs, or what the hardware can afford), and this appendix sorts them out. Lesson 3 defined the one operation everything here is a variant of — scaled dot-product attention,\u003c/p\u003e","title":"Appendix C — Attention Variants"},{"content":"Appendix B — Cheat Sheets: How the Architectures Compare This appendix is a reference. Where the chapters told the story of each architecture, here we lay two of the most important contrasts side by side, in plain comparisons you can return to when you need to decide which tool fits a job. Two comparisons matter most: the reader against the writer, and the dense model against the mixture-of-experts.\nReader versus writer The one idea everything flows from Every difference between a reader (an encoder, Chapter 5 ) and a writer (a decoder, Chapter 4 ) traces back to a single design choice: what is the model allowed to look at? A writer may only look backward, at what it has already produced or been given, never at what comes next. A reader may look in both directions at once, at the whole passage. That one rule cascades into everything else. Because the writer can only see backward, it can be trained to guess the next word and then actually produce text. Because the reader can see everything, it can build deep understanding — but it loses the ability to guess the next word, since the answer is sitting right there in front of it.\nThe differences at a glance Question Reader (encoder) Writer (decoder) What does it look at? The whole passage, both directions Only what came before, left to right What does it produce? Understanding — internal representations New text, one word at a time How is it trained? Fill-in-the-blank Guess the next word Can it generate text? No Yes — that is its purpose How is it adapted to a task? Add a small task head, train on labeled data Instruction-tune, align, or just prompt Does adapting need labeled data? Usually yes Often no Speed pattern One fast pass A slower word-by-word loop Best at Understanding, sorting, matching, search Writing, answering, conversing A few of these deserve a word. The reader does its work in a single pass, reading everything in parallel, which is why it is cheap and fast for understanding tasks. The writer must loop, producing text one word at a time, each new word requiring another pass — inherently slower. And the two are adapted in fundamentally different ways: a reader is steered by supervision (attach a head, train on labeled examples), while a writer is steered by behavior (describe the task, show a couple of examples, or shape it as in Chapter 10 ), often with no task-specific data at all.\nWhich should I reach for? Walk through three questions.\nFirst: do you need the system to produce new text? If no — you want to classify, tag, extract, search, or match — reach for a reader. Sentiment analysis, spam detection, search relevance, document tagging all live here. If yes, keep going.\nSecond: are you transforming one whole given input into an output — translating it, summarizing it, rewriting it? If yes, reach for a reader-plus-writer combination that reads the input deeply and then writes the output. If it is free-form instead — open chat, completions, answering questions — reach for a writer on its own.\nThird: are you building search or a retrieval-augmented system? Then you will often want both: a reader to understand and match documents to a query, and a writer to compose the final answer from what was retrieved. The reader finds and understands; the writer speaks. They are complements, not competitors.\nFigure 1. One rule, and everything that cascades from it.\nDense versus mixture-of-experts The one idea everything flows from A dense model uses all of itself to process every word. A mixture-of-experts model (Chapter 7 ) keeps a large bank of expert sub-modules but, for each word, switches on only a small handful, chosen by a little router. That single change — \u0026ldquo;use everything every time\u0026rdquo; versus \u0026ldquo;use a chosen few each time\u0026rdquo; — drives every other difference.\nThe differences at a glance Aspect Dense model Mixture-of-experts How much is used per word? All of it A chosen few experts Size numbers to track One Two: total (memory) and active (work) Work per word Grows with the full size Tracks only the small active slice Memory footprint Matches its size Large — all experts stay loaded A router deciding per word? No Yes Special training need None extra Must keep the experts balanced Main bottleneck Raw computation Machine-to-machine shuffling (training); memory (serving) Attention and output controls Standard Identical to dense Fine-tuning risk Standard More prone to overfitting The headline is the two size numbers. A dense model has one: how much it knows and how much work it does per word are the same figure, welded together. A mixture-of-experts model splits them — a large total size that sets how much it can know and how much memory it occupies, and a small active size that sets how much work each word costs. This is the whole trick: it decouples \u0026ldquo;how much it knows\u0026rdquo; from \u0026ldquo;how much it costs to run per word,\u0026rdquo; so you can grow the total enormously while keeping the per-word cost modest. The catch is memory: all the experts must stay loaded even though each word uses only a few, so a mixture-of-experts model has a small model\u0026rsquo;s compute but a large model\u0026rsquo;s memory.\nThe router is the new piece, and it brings problems a dense model never faces: the lazy-favoritism collapse, the consequent need for active load balancing, more knobs to set, and extra fragility that demands careful handling. It is also worth stressing what stays identical: the attention mechanism, the caching used during generation, and all the dials that control output. The entire mixture-of-experts trick lives inside one sub-layer; from the outside, the model behaves like any other.\nWhen should I reach for a mixture-of-experts model? Start with memory, because it is the deciding factor. Is memory tight — do you need to fit on just a few devices? If memory is your binding constraint, stay dense: a mixture-of-experts model would force you to hold all the experts in memory anyway, so its main advantage (cheap per-word compute) does not help you when memory is exactly what you have run out of. With a dense model, your capacity is simply whatever you can hold.\nIf you have memory to spare, ask: do you want more quality without paying more compute per word? If yes, this is the sweet spot for a mixture-of-experts model — spend memory to buy extra capacity and quality while keeping the per-word running cost flat. If instead simplicity matters more to you, stay dense and avoid the router, the balancing machinery, and the cross-machine shuffling entirely.\nThe compressed rule: reach for a mixture-of-experts model when you have memory to spend and want higher quality at low per-word cost; stay dense when memory is the binding constraint, or when operational simplicity is worth more than the compute savings.\nFigure 2. Use everything every time, or use a chosen few each time.\nFigure 3. Start with memory, because it is the deciding factor.\n← Appendix A · Field Guide to Real Models · Contents · Appendix C · Attention Variants → The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/tutorials/inside-the-machines/appendix-b-comparison-cheat-sheets/","summary":"\u003ch1 id=\"appendix-b--cheat-sheets-how-the-architectures-compare\"\u003eAppendix B — Cheat Sheets: How the Architectures Compare\u003c/h1\u003e\n\u003cp\u003eThis appendix is a reference. Where the chapters told the story of each architecture, here we lay two of the most important contrasts side by side, in plain comparisons you can return to when you need to decide which tool fits a job. Two comparisons matter most: the reader against the writer, and the dense model against the mixture-of-experts.\u003c/p\u003e\n\u003ch2 id=\"reader-versus-writer\"\u003eReader versus writer\u003c/h2\u003e\n\u003ch3 id=\"the-one-idea-everything-flows-from\"\u003eThe one idea everything flows from\u003c/h3\u003e\n\u003cp\u003eEvery difference between a reader (an encoder, \u003ca href=\"/tutorials/inside-the-machines/chapter-05-encoder-models/\"\u003eChapter 5\u003c/a\u003e\n) and a writer (a decoder, \u003ca href=\"/tutorials/inside-the-machines/chapter-04-decoder-models/\"\u003eChapter 4\u003c/a\u003e\n) traces back to a single design choice: \u003cem\u003ewhat is the model allowed to look at?\u003c/em\u003e A writer may only look \u003cem\u003ebackward\u003c/em\u003e, at what it has already produced or been given, never at what comes next. A reader may look in \u003cem\u003eboth directions\u003c/em\u003e at once, at the whole passage. That one rule cascades into everything else. Because the writer can only see backward, it can be trained to guess the next word and then actually produce text. Because the reader can see everything, it can build deep understanding — but it loses the ability to guess the next word, since the answer is sitting right there in front of it.\u003c/p\u003e","title":"Cheat Sheets: How the Architectures Compare"},{"content":"Appendix C — A Field Guide to Attention Variants Chapter 3 introduced attention as the meeting where every word glances around at the others and decides which ones matter. That one mechanism is the whole game — but in real models it comes in several outfits, and the names can pile up into intimidating alphabet soup. This appendix is the decoder ring. The trick is to see that every \u0026ldquo;attention variant\u0026rdquo; is just that same meeting run under different rules, and that the rules answer practical questions: what does the task require, and what can the hardware afford? It helps to picture three independent dials — who is allowed to look at whom, how much memory the looking costs while writing, and how to keep the looking affordable when the text gets very long. Let us turn each in turn.\nDial one: who is allowed to look at whom The most important choice is not about saving money — it is forced by the job. There are three settings.\nLooking only backward (causal). When a model is generating text one word at a time, each word may look only at the words before it, never after. The reason is almost obvious once said aloud: the model\u0026rsquo;s job is to predict the next word, and letting it peek at the future would be cheating, like a student seeing the answer key. So every word wears blinders blocking the road ahead. This backward-only style is the heart of every chatbot-style writer, and it is exactly the rule from Chapter 4 .\nLooking both ways (bidirectional). Sometimes the whole text is already present and nothing needs to be generated — you just want it understood, as when classifying a review or building a search index. There is no future to protect, so every word may look in both directions and draw on everything around it. This is the style behind the reader models of Chapter 5 , and it is also how models read images that have been chopped into patches. It is wonderful for comprehension and useless for generation, because the moment a word can see the future, next-word prediction collapses.\nLooking at a different text (cross-attention). The third setting has words in one text look over at a second text entirely — the mechanism of \u0026ldquo;pay attention to that other thing while you work,\u0026rdquo; and the one new idea behind the encoder–decoder models of Chapter 6 . The classic case is translation: as the model writes the French sentence, each French word glances back at the English original. The same trick lets a text model look at an image, so the words being generated can attend to what is actually in the picture. Modern all-in-one chatbots mostly skip it by stuffing everything into one long prompt, but it remains the natural tool whenever there are genuinely two separate sources to reconcile.\nFigure 1. Dial one: three settings, and the task picks for you.\nDial two: sharing notes to save memory The remaining variants exist purely to make attention cheaper, and the first cost problem appears while the model is writing. Recall from Chapter 3 that a model runs many attention meetings in parallel, called heads, each specializing in a different kind of relationship. Many heads help quality. The trouble is that, to avoid redoing work, the model keeps a running notebook of what each head has computed about every word so far (this is the cache from Chapter 11 ). With many independent heads, that is many separate notebooks, and as a conversation grows long the notebooks balloon until memory, not arithmetic, becomes the bottleneck.\nThe fix is delightfully simple: let the heads share notes. Instead of every head keeping its own private record of what each word is and offers, many heads consult one shared record. The most aggressive version has all heads share a single common notebook — multi-query attention — which shrinks memory dramatically but, by sharing so hard, blurs the heads\u0026rsquo; individuality and dips quality a little. So the field settled on a middle ground, grouped-query attention, now the standard: picture the heads sorted into a few small teams, everyone on a team sharing one notebook but different teams keeping separate ones. You get almost all the memory savings of full sharing while keeping almost all the quality of full independence, which is why most modern open models use it. (A newer, fancier idea compresses the notebook into a tight summary and expands it on demand, pushing the savings further, but it is more complex and not yet everywhere.) The thing to remember: none of these tricks change what the model fundamentally computes; they shrink the temporary scratch memory used during generation, trading a sliver of quality for the ability to serve long conversations affordably.\nFigure 2. Dial two: how many notebooks the heads keep.\nDial three: keeping it affordable on long texts The second cost problem is different. Run attention in full and every word looks at every other word, so doubling the length of the text roughly quadruples the work — nothing for a paragraph, brutal for a whole book. This is the quadratic price tag flagged back in Chapter 3. Several tricks tame it.\nOnly look at nearby words (sliding window). The simplest idea: rather than surveying the entire text, each word looks only at a window of its recent neighbors — the way you can follow a conversation by tracking the last few sentences instead of re-reading the whole transcript. The cost now grows gently with length instead of exploding. And because the model is many layers deep, information still travels far: a word sees its neighbors, who saw their neighbors, so news ripples across the whole text over several layers even though no single layer looks far. The tradeoff is that truly long-range links get weaker, which is why designs often mix in occasional full-view layers.\nOnly look at a chosen few (sparse attention). A relative of the window: instead of looking at everything, each word looks at a carefully chosen sprinkling of others — some neighbors, plus a few \u0026ldquo;anchor\u0026rdquo; words everyone may consult. It is efficient and clever, but the patterns are hand-designed and a bit brittle, so it never became mainstream for chatbots.\nThe clever bookkeeping trick (FlashAttention). This one is worth highlighting because it is not an approximation. It computes exactly the same attention as the plain version — identical result, no corners cut — but far faster. The insight is purely about how computers handle memory. Naively, the model builds a giant table of every-word-against-every-word, parks it in slow memory, then reads it back, and that shuffling to and from slow memory is the real bottleneck, not the arithmetic. The trick does the work in small tiles right inside the chip\u0026rsquo;s fast scratch memory and never writes the giant table out at all. Same math, dramatically less memory traffic, much quicker. Its arrival was so effective that it largely removed the pressure to use the approximate shortcuts above — the field mostly returned to full, exact attention because smarter bookkeeping made it cheap enough.\nFigure 3. Dial three: how much of the text each word actually sees.\nThe takeaway Attention is one idea wearing several outfits. One dial sets who may look at whom, and that one is dictated by the task — backward-only for writing, both-ways for understanding, across for reconciling two sources. The other dials are about cost: heads sharing notebooks to shrink the memory used while generating, and windows, sparse patterns, or smarter bookkeeping to keep the looking affordable on long texts. None of it changes the soul of the mechanism from Chapter 3; it just makes that mechanism fit the job and the hardware.\n← Appendix B · Comparison Cheat Sheets · Contents · Appendix D · Where Words Sit → The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/tutorials/inside-the-machines/appendix-c-attention-variants/","summary":"\u003ch1 id=\"appendix-c--a-field-guide-to-attention-variants\"\u003eAppendix C — A Field Guide to Attention Variants\u003c/h1\u003e\n\u003cp\u003e\u003ca href=\"/tutorials/inside-the-machines/chapter-03-attention/\"\u003eChapter 3\u003c/a\u003e\n introduced attention as the meeting where every word glances around at the others and decides which ones matter. That one mechanism is the whole game — but in real models it comes in several outfits, and the names can pile up into intimidating alphabet soup. This appendix is the decoder ring. The trick is to see that every \u0026ldquo;attention variant\u0026rdquo; is just that same meeting run under different rules, and that the rules answer practical questions: what does the task require, and what can the hardware afford? It helps to picture three independent dials — who is allowed to look at whom, how much memory the looking costs while writing, and how to keep the looking affordable when the text gets very long. Let us turn each in turn.\u003c/p\u003e","title":"A Field Guide to Attention Variants"},{"content":"Appendix D — Positional Encodings Technical course on LLM mechanics — Appendix to Lesson 2 (§4, \u0026ldquo;Injecting position\u0026rdquo;).\nA catalogue of the ways a transformer is told where each token sits. This is a side-quest off Lesson 2 §4: that section introduced three schemes in passing because position is one of the few model components that is genuinely a design choice rather than a forced consequence of the math, and the choice has quietly become one of the most active areas of LLM engineering. This appendix collects every family that matters in practice, with the math, the property that defines each, and where it is [CONFIG] (chosen, no parameters) versus [LEARNED] (trained).\nEquations render with any LaTeX-aware Markdown viewer (VS Code + Markdown+Math, Obsidian, or GitHub).\n← Course index · Lesson 2 — Tokenization \u0026amp; Embeddings · Appendix B — Formulas Why position has to be injected at all Self-attention (Lesson 3) is permutation-equivariant: permute the input rows and the output rows permute identically. The dot product $\\mathbf q_i^\\top\\mathbf k_j$ depends on which vectors meet, never on their indices $i$ and $j$. So a bare transformer cannot tell \u0026ldquo;dog bites man\u0026rdquo; from \u0026ldquo;man bites dog\u0026rdquo; — to it they are the same multiset of tokens. Order is information the model would otherwise throw away, so it must be supplied explicitly.\nEvery scheme below answers one question: how do we make the attention score $s_{ij}$ depend on the positions $i$ and $j$? They differ along three axes:\nAbsolute vs. relative. Does a token learn where it is (position $i$), or only how far it is from another token (offset $i-j$)? Relative information is what language mostly needs (\u0026ldquo;the adjective two words back\u0026rdquo;), and it extrapolates to lengths never seen in training. Where it is applied. Added to the embedding before attention (input-level), injected into the dot product (score-level), or added as a bias to the score (additive). Learned vs. fixed. A trainable table ([LEARNED], costs parameters, usually caps the context length) or a closed-form function ([CONFIG], no parameters, often extrapolates). With that question fixed, the families below are best read as a progression: each one is a response to a limitation of the one before, and the table previews where every scheme lands on the three axes.\nThe families, at a glance:\nFamily Representative Abs/Rel Applied to Params None NoPE — (implicit) nothing [CONFIG] Absolute, fixed Sinusoidal Absolute embedding (add) [CONFIG] Absolute, learned Learned table (GPT-2, BERT) Absolute embedding (add) [LEARNED] Relative, learned Shaw et al.; Transformer-XL Relative keys / scores [LEARNED] Relative, bias T5 bucket bias Relative score (add) [LEARNED] (scalars) Relative, bias, fixed-slope ALiBi Relative score (add) [CONFIG] Rotary RoPE Relative Q \u0026amp; K (rotate) [CONFIG] Rotary, extended PI · NTK · YaRN Relative Q \u0026amp; K (rotate) [CONFIG] Contextual CoPE Relative (content-based) keys [LEARNED] Figure 1. Every family, on the three axes that separate them.\n1. No positional encoding (NoPE) The null option: inject nothing, and let a causal, decoder-only model infer position on its own. The trick is that the causal mask already breaks permutation symmetry — token $i$ can see exactly $i$ predecessors, so the count of visible tokens is itself a position signal that the network can read out through depth.\n$$ s_{ij}=\\frac{\\mathbf q_i^\\top\\mathbf k_j}{\\sqrt{d_k}}\\quad\\text{(no positional term at all).} $$Surprisingly, NoPE decoders train and can match explicit schemes at small scale, and they extrapolate to longer sequences reasonably well. It does not work for bidirectional (encoder) models, where the mask is symmetric and the symmetry is never broken. Mostly a research baseline, but a clarifying one: it shows position can be implicit. [CONFIG], zero parameters.\n2. Absolute sinusoidal (fixed) The original \u0026ldquo;Attention Is All You Need\u0026rdquo; scheme. To the embedding at position $p$, add a fixed vector whose coordinate pairs rotate at geometrically spaced frequencies:\n$$ PE_{(p,2k)}=\\sin\\!\\big(p\\,/\\,10000^{2k/d}\\big),\\qquad PE_{(p,2k+1)}=\\cos\\!\\big(p\\,/\\,10000^{2k/d}\\big). $$Wavelengths run from $2\\pi$ up to $\\sim\\!10000\\cdot2\\pi$, so low dimensions track fine position and high dimensions track coarse position — every $p$ gets a unique signature, like a binary place-value code in continuous form. The defining property: $PE_{p+\\Delta}$ is a fixed linear function of $PE_p$ (a rotation by $\\Delta\\omega_k$ per pair), so a linear attention map can in principle recover relative offsets. In theory it extrapolates to unseen lengths; in practice the extrapolation is weak. [CONFIG], no parameters.\n3. Absolute learned Replace the closed form with a trainable table $P_{\\text{pos}}\\in\\mathbb R^{n_{\\max}\\times d}$, one row per position, added to the embedding:\n$$ H^{(0)}_p = \\mathbf x_p + P_{\\text{pos},\\,p}. $$Used by GPT-2 and BERT. Simplest to implement and lets the model fit whatever positional structure the data shows. Two costs: it adds $n_{\\max}\\,d$ parameters, and it cannot extrapolate — there is no row for position $n_{\\max}+1$, so the context length is a hard architectural cap. [LEARNED].\n4. Relative position, learned (Shaw et al., 2018) The first move away from absolute coordinates. Instead of tagging each token with where it is, add a learned vector that depends on the offset $i-j$ between query $i$ and key $j$, injected inside the score (and optionally the value sum):\n$$ s_{ij}=\\frac{\\mathbf q_i^\\top\\big(\\mathbf k_j+\\mathbf a^{K}_{\\,i-j}\\big)}{\\sqrt{d_k}}, $$where $\\{\\mathbf a^{K}_r\\}$ is a learned table indexed by clipped relative distance $r=\\mathrm{clip}(i-j,-R,R)$. Clipping to a window $R$ keeps the table finite and lets any offset beyond $R$ share an embedding, which is what allows generalization to longer sequences. This is the conceptual ancestor of every relative scheme that follows. [LEARNED].\nA close cousin, Transformer-XL (Dai et al., 2019), rewrites the expanded score $\\mathbf q_i^\\top\\mathbf k_j$ into four terms and replaces the absolute-position parts with a sinusoidal relative encoding plus two learned global bias vectors $\\mathbf u,\\mathbf v$:\n$$ s_{ij}\\;\\propto\\;\\mathbf q_i^\\top\\mathbf k_j \\;+\\; \\mathbf q_i^\\top R_{\\,i-j}\\;+\\;\\mathbf u^\\top\\mathbf k_j\\;+\\;\\mathbf v^\\top R_{\\,i-j}, $$with $R_{\\,i-j}$ a fixed sinusoid of the offset. This made segment-recurrent (\u0026ldquo;XL\u0026rdquo;) attention over long documents work and directly influenced T5 and RoPE.\n5. Relative bias — T5 T5 strips relative position down to its cheapest possible form: a single learned scalar added to the attention logit, depending only on the offset, with no interaction with the content vectors:\n$$ s_{ij}=\\frac{\\mathbf q_i^\\top\\mathbf k_j}{\\sqrt{d_k}} + b_{\\,\\beta(i-j)}. $$Offsets are mapped into a small number of buckets $\\beta(\\cdot)$ — exact for nearby tokens, logarithmically coarsening for distant ones — and each bucket holds one learned scalar $b$, shared across all layers\u0026rsquo; positions but separate per head. A few dozen parameters per head buy a flexible, length-extrapolating bias. Widely used (T5, and the template for many later bias schemes). [LEARNED] (a handful of scalars).\n6. ALiBi — Attention with Linear Biases ALiBi (Press et al., 2021) keeps the additive-bias idea but makes the bias a fixed, non-learned linear penalty on distance: the farther back a key is, the more its score is reduced.\n$$ s_{ij}=\\frac{\\mathbf q_i^\\top\\mathbf k_j}{\\sqrt{d_k}} \\;-\\; m_h\\,(i-j),\\qquad j\\le i, $$where $m_h$ is a head-specific slope fixed in advance (a geometric sequence, e.g. $m_h=2^{-8h/H}$ across $H$ heads). Different heads therefore attend at different effective ranges — steep slopes stay local, shallow slopes see far. Because the bias is just a function of distance with no length-dependent parameters, ALiBi extrapolates strongly: a model trained at length 1k runs at much longer lengths with little degradation. No embeddings added, no parameters learned. [CONFIG].\n7. RoPE — Rotary Position Embedding (today\u0026rsquo;s default) RoPE (Su et al., 2021) injects position not by adding anything, but by rotating the query and key vectors by an angle proportional to their absolute position, in 2-D coordinate pairs. For pair $k$ at position $p$, with frequency $\\omega_k=10000^{-2k/d}$:\n$$ \\begin{pmatrix}q'_{2k}\\\\ q'_{2k+1}\\end{pmatrix} =\\begin{pmatrix}\\cos p\\omega_k \u0026 -\\sin p\\omega_k\\\\ \\sin p\\omega_k \u0026 \\cos p\\omega_k\\end{pmatrix} \\begin{pmatrix}q_{2k}\\\\ q_{2k+1}\\end{pmatrix}, $$and the same rotation by angle $m\\omega_k$ is applied to the keys at position $m$. The defining identity is what makes it special: because rotations compose by adding angles, the resulting dot product depends on the two positions only through their difference:\n$$ \\langle q'_p,\\,k'_m\\rangle=\\sum_k \\big(q_{2k}k_{2k}+q_{2k+1}k_{2k+1}\\big)\\cos\\!\\big((p-m)\\omega_k\\big)+(\\text{sine cross-terms}), $$i.e. $\\langle q'_p,k'_m\\rangle = g(\\mathbf q,\\mathbf k,\\,p-m)$. So RoPE is applied with absolute positions but yields a relative score — the best of both worlds. It adds no parameters, plays well with the KV cache (each key is rotated once, by its own position), and is the scheme in LLaMA, GPT-NeoX, PaLM, Qwen, Mistral, and most current models. Frequencies are [CONFIG].\nFigure 2. One position, several frequencies.\n8. Extending RoPE to longer contexts A model trained with RoPE at context $L_{\\text{train}}$ degrades past $L_{\\text{train}}$ because the high-frequency pairs complete many rotations and the model never saw those phase values. Three closely related tricks rescale the angles so a longer context \u0026ldquo;looks like\u0026rdquo; the trained one — all are [CONFIG], applied at fine-tune or even inference time, no new parameters.\nPosition Interpolation (PI). Linearly squeeze positions back into the trained range by a factor $L_{\\text{train}}/L_{\\text{new}}$:\n$$ p \\;\\longmapsto\\; p\\cdot\\frac{L_{\\text{train}}}{L_{\\text{new}}}, $$so every angle stays within the range seen in training. Simple and effective with a short fine-tune; the cost is some loss of fine-grained local resolution.\nNTK-aware scaling. Instead of scaling all frequencies equally, change the base $10000\\!\\to\\!10000\\cdot s$ so that high frequencies (local detail) are barely touched while low frequencies (long-range) are stretched — preserving local precision better than PI, often with no fine-tuning at all.\nYaRN. Combines NTK-style per-frequency scaling with a small attention-temperature correction, interpolating wavelengths by band (leave short waves alone, interpolate long waves, blend the middle). The current state of the art for large context extension, reaching very long contexts with minimal fine-tuning.\n9. Other and emerging schemes A few more, useful to recognize:\nCoPE — Contextual Position Encoding (Golovneva et al., 2024). Position is measured not by raw token count but by a content-gated cumulative sum: the model decides, per query, which previous tokens \u0026ldquo;count\u0026rdquo; toward distance (e.g. count words, or sentences, or none). This lets position track semantic units rather than tokens. [LEARNED]. Kerple / Sandwich / FIRE. Variations on additive relative bias — Kerple generalizes ALiBi with learnable kernels, FIRE passes the offset through a small learned MLP for a fully functional, length-generalizing bias. [LEARNED]. Sinusoidal relative (in Transformer-XL/§4). A fixed sinusoid of the offset rather than the absolute position; bridges §2 and §4. How to choose For a decoder-only LLM today, RoPE (§7) is the default, with YaRN/NTK (§8) layered on when you need to extend context beyond the trained length — this is what most open models ship. ALiBi (§6) is the main alternative when raw length-extrapolation out of the box is the priority and you can accept its always-local bias. Learned absolute (§3) survives mainly in older encoders (BERT) and is fine when the context length is fixed and modest. T5 relative bias (§5) remains standard in encoder–decoder models. Sinusoidal (§2) is now largely of historical and pedagogical interest, and NoPE (§1) is a research baseline.\nThe throughline mirrors the rest of the course: the field moved from absolute position added at the input (costs parameters, caps length) toward relative position injected into the score (no parameters, extrapolates) — and RoPE\u0026rsquo;s trick of getting a relative score from an absolute rotation is why it won.\nFigure 3. The field\u0026rsquo;s move from absolute to relative.\n← Course index · Lesson 2 — Tokenization \u0026amp; Embeddings · Appendix B — Formulas The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/tutorials/llm-lessons/appendix-d-positional-encodings/","summary":"\u003ch1 id=\"appendix-d--positional-encodings\"\u003eAppendix D — Positional Encodings\u003c/h1\u003e\n\u003cp\u003e\u003cem\u003eTechnical course on LLM mechanics — Appendix to Lesson 2 (§4, \u0026ldquo;Injecting position\u0026rdquo;).\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eA catalogue of the ways a transformer is told \u003cstrong\u003ewhere\u003c/strong\u003e each token sits. This is a side-quest off Lesson 2 §4: that section introduced three schemes in passing because position is one of the few model components that is genuinely a design choice rather than a forced consequence of the math, and the choice has quietly become one of the most active areas of LLM engineering. This appendix collects every family that matters in practice, with the math, the property that defines each, and where it is \u003ccode\u003e[CONFIG]\u003c/code\u003e (chosen, no parameters) versus \u003ccode\u003e[LEARNED]\u003c/code\u003e (trained).\u003c/em\u003e\u003c/p\u003e","title":"Appendix D — Positional Encodings"},{"content":"Appendix D — Telling a Model Where Words Sit Chapter 2 mentioned, almost in passing, that a model has to be told the order of its words. This appendix unpacks that, because the question of how to inform a model about word order turns out to have a small, elegant history of its own — and it explains one of the quiet reasons modern models can handle ever-longer texts.\nWhy order has to be spelled out at all Start with a quirk that surprises people: the attention mechanism, left to itself, is blind to word order. It sees a sentence as a bag of words with no sense of sequence — so \u0026ldquo;dog bites man\u0026rdquo; and \u0026ldquo;man bites dog\u0026rdquo; look identical to it, the same words tossed in a bag. But those sentences mean opposite things, and the difference lives entirely in the order. Since order carries real meaning, and the core machinery ignores it, someone has to hand the model the seating chart explicitly. Every scheme below is just a different way of telling each word where it sits. They run from clumsy to elegant, and each was invented to fix a flaw in the one before.\nGiving every word a seat number The first idea is the obvious one: stamp each word with its seat number — first, second, third — and let the model learn what each seat tends to mean. This works fine and is easy. But it has a stubborn weakness. If the model only ever trained on texts up to, say, a thousand words, it has never seen a seat numbered one-thousand-and-one. Confronted with a longer text, it simply has no stamp for those later seats and falls apart. The seating chart was printed for a fixed-size room, and the model cannot cope when the room turns out to be bigger than any it trained in. That brittleness at unfamiliar lengths is the flaw everything afterward tries to repair.\nEncoding the gaps, not the seats The key realization is that what language actually cares about is rarely the absolute seat number — it is the distance between words. \u0026ldquo;The adjective two words before this noun\u0026rdquo;: that \u0026ldquo;two words before\u0026rdquo; is what matters, and it is true whether the phrase lands at the start of a sentence or the end. So instead of stamping each word with where it sits, you encode how far apart any two words are. This fits how language works much better, and it carries a lovely bonus: a distance like \u0026ldquo;three apart\u0026rdquo; or \u0026ldquo;ten apart\u0026rdquo; means the same thing no matter how long the overall text is, so a model built this way handles lengths it never trained on far more gracefully. It learned the gaps, not the seat numbers, and gaps do not run out.\nFigure 1. Seat numbers run out; gaps do not.\nGently penalizing distance One especially clean version of the relative idea barely \u0026ldquo;encodes\u0026rdquo; anything at all. It simply applies a soft penalty: the farther back a word is, the less weight it gets — attention quietly fades with distance, the way your memory of a conversation grows fuzzier the further back you reach. Nothing has to be learned or stored; it is a fixed, built-in lean toward recent words, with different attention heads set to fade at different rates so some stay tightly local while others reach further. Because it is just a simple function of distance with no fixed-size table behind it, a model using it can be trained on short texts and still run sensibly on much longer ones straight out of the box.\nRotating the words: today\u0026rsquo;s favorite The modern winner is a genuinely clever maneuver. Picture each word\u0026rsquo;s representation as a little arrow on a dial. Before the words compare themselves to one another, you rotate each word\u0026rsquo;s arrow by an amount that depends on its position — the first word turned a little, the tenth word turned ten times as much, and so on. The magic is what happens when two rotated words then compare: because rotations simply add up, the comparison ends up depending only on the difference in their rotations — that is, on how far apart the two words are. So even though each word was rotated by its own absolute position, what comes out is purely about relative distance. You get the simplicity of stamping each word independently and the robustness of caring only about gaps, all in one move. This is why it became the default in most current models: it is clean, it costs nothing to store, and it naturally cares about the right thing. (This is the \u0026ldquo;modern trick\u0026rdquo; Chapter 2 alluded to.)\nFigure 2. Rotate each word by its own position; only the gap survives.\nStretching a model to longer texts There is a final, very practical bonus, and it is the reason this appendix matters beyond curiosity. A model trained on short texts with the rotation scheme can be coaxed into handling much longer ones by adjusting the dial settings. The intuition: if you trained in a small room and now face a much bigger one, you can gently compress the new seating so the far-apart words appear closer together — back inside the range of distances the model already understands — or you can selectively rescale the rotations so fine local detail stays sharp while only the long-range spacing gets stretched. With a touch of extra training, sometimes none at all, a model built for short texts can be expanded to read very long ones. It is effectively re-spacing the seats so an unfamiliar, oversized room looks like the familiar one the model grew up in. This is much of the machinery behind the headline \u0026ldquo;this model now supports a context of hundreds of thousands of words.\u0026rdquo;\nThe takeaway Word order is meaning, but attention is blind to it, so order must be supplied. The history runs from stamping absolute seat numbers (simple but brittle at new lengths), to encoding the gaps between words (robust, because gaps don\u0026rsquo;t run out), to gently fading attention with distance, to the modern favorite of rotating each word by its position so that only relative distance survives the comparison. And because the best of these schemes care about distance rather than absolute position, they can be stretched — letting a model trained on short texts grow to read very long ones.\nFigure 3. From seat numbers to rotations, each fixing the last.\n← Appendix C · Attention Variants · Contents · Appendix E · The Chinchilla Story → The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/tutorials/inside-the-machines/appendix-d-positional-encodings/","summary":"\u003ch1 id=\"appendix-d--telling-a-model-where-words-sit\"\u003eAppendix D — Telling a Model Where Words Sit\u003c/h1\u003e\n\u003cp\u003e\u003ca href=\"/tutorials/inside-the-machines/chapter-02-from-words-to-numbers/\"\u003eChapter 2\u003c/a\u003e\n mentioned, almost in passing, that a model has to be told the order of its words. This appendix unpacks that, because the question of how to inform a model about word order turns out to have a small, elegant history of its own — and it explains one of the quiet reasons modern models can handle ever-longer texts.\u003c/p\u003e\n\u003ch2 id=\"why-order-has-to-be-spelled-out-at-all\"\u003eWhy order has to be spelled out at all\u003c/h2\u003e\n\u003cp\u003eStart with a quirk that surprises people: the attention mechanism, left to itself, is \u003cem\u003eblind to word order\u003c/em\u003e. It sees a sentence as a bag of words with no sense of sequence — so \u0026ldquo;dog bites man\u0026rdquo; and \u0026ldquo;man bites dog\u0026rdquo; look identical to it, the same words tossed in a bag. But those sentences mean opposite things, and the difference lives entirely in the order. Since order carries real meaning, and the core machinery ignores it, someone has to hand the model the seating chart explicitly. Every scheme below is just a different way of telling each word \u003cem\u003ewhere it sits.\u003c/em\u003e They run from clumsy to elegant, and each was invented to fix a flaw in the one before.\u003c/p\u003e","title":"Telling a Model Where Words Sit"},{"content":"Appendix E — Compute-Optimal Scaling: The Chinchilla Story Chapter 9 touched on the question of how to split a training budget between a bigger model and more data, and named the Chinchilla correction. This appendix tells that story in full, because it is one of the most illuminating episodes in the field — a case where the conventional wisdom was confidently wrong, got corrected by a single careful study, and then turned out to have a twist that the correction itself missed.\nThe surprise that started it: quality you can forecast One of the strangest and most useful discoveries in modern AI is that the quality of a language model does not improve in fits and starts as you make it bigger and train it longer. It improves smoothly — and, crucially, predictably. Run a handful of small, cheap experiments, plot how good the models get as you spend more, and the line is so regular that you can extend it forward and estimate, with real confidence, how good a model ten or a hundred times larger will be, before you have spent a penny building it.\nThis is a remarkable thing to be able to do. In most of engineering you cannot test a paper airplane and confidently predict a jumbo jet. But in this corner of AI the small experiments really do tell you about the big ones. That predictability is what turned model-building from guesswork into planning. The central question stops being \u0026ldquo;will this work?\u0026rdquo; and becomes \u0026ldquo;given what I am willing to spend, what is the smartest way to spend it?\u0026rdquo;\nFigure 1. Run small experiments, extend the line, forecast the big model.\nThe fixed-budget choice: a bigger brain or more reading Here is the trade-off at the heart of everything. Building a model costs a fixed pot of compute — think of it as a fixed number of hours on expensive machines. With that pot you face one unavoidable choice: spend it on a bigger model (more internal knobs, more raw capacity) or on more training data (showing the model far more text). You cannot maximize both — every unit that buys a bigger brain is a unit that cannot buy more reading.\nThe cleanest analogy is a student with a fixed number of study-hours before an exam. They can grow a bigger memory — more raw mental capacity — or spend those same hours reading more material. A huge memory stuffed with very little reading is wasted potential; a tiny memory drowning in more books than it can absorb is also wasted. Somewhere in between is the sweet spot, and the whole game is finding it.\nFigure 2. One fixed pot, three ways to split it.\nThe early advice: mostly just make it bigger When researchers first worked this out, around 2020, their answer was lopsided: when you get more compute, pour most of it into making the model bigger, and only a little into extra data. Build a giant brain; do not worry too much about over-feeding it.\nThe field took this to heart, and the result was a generation of famous models that were enormous but, in hindsight, badly underfed — giant brains that had simply not read enough. The best-known example was a model of around a hundred and seventy-five billion knobs trained on a relatively thin diet of text — only a couple of words of training data per knob. Picture a student with an extraordinary memory who skimmed only a couple of textbooks: impressive raw capacity, badly underused. These models were genuinely good and genuinely groundbreaking, but they were leaving performance on the table, and nobody quite realized how much.\nThe Chinchilla correction: grow them together Then came the study that reset the field\u0026rsquo;s thinking, nicknamed after its model, Chinchilla. The researchers re-ran the whole experiment more carefully, and fixed a subtle flaw: in the earlier work, the data-heavy experiments had been quietly handicapped by a technical detail in how training was scheduled, which made \u0026ldquo;more data\u0026rdquo; look less valuable than it really was. Once that was corrected, the picture changed completely.\nThe new conclusion was a model of balance: model size and training data should grow roughly in step with each other. Don\u0026rsquo;t lavish everything on the brain and starve it of reading — scale both together. This crystallized into the single most quoted number in the field: roughly twenty words of training data for every knob in the model. It is a back-of-the-napkin rule, and that is exactly why people love it — it turns a hard planning problem into a one-line estimate.\nThe team then delivered the knockout. They built a model four times smaller than a rival flagship but, using the same compute budget, spent the savings on feeding it far more text — and the smaller, better-read model beat the larger, underfed one across the board. Back to the student: a kid with a more modest memory who actually did the reading outscored the prodigy who had barely cracked a book, and both spent the same hours studying. That single result was the proof, and after it \u0026ldquo;balance size and data\u0026rdquo; became the field\u0026rsquo;s default instinct.\nThe twist nobody saw coming: training-optimal is not serving-optimal Here the story takes its most important real-world turn, because everything above is about getting the best model for a fixed training budget. It says nothing about what happens after the model is built, when it is actually being used — and that distinction turns out to be everything.\nA trained model gets run to answer questions, write text, and power products — potentially billions of times. And every one of those runs costs more for a bigger model. A bigger brain is not just more expensive to build; it is more expensive to operate, forever, with every query it ever answers. So if you know a model will be used heavily and for a long time, the math flips. It becomes worth deliberately building a smaller model and then over-training it — feeding it far more than the tidy twenty-words-per-knob rule suggests, well past the point that is efficient for training. You knowingly \u0026ldquo;waste\u0026rdquo; training money to end up with a small model that is permanently cheap to run, yet still genuinely smart because it read so much.\nThis is the philosophy behind the famous Llama-style models: small brains trained on staggering amounts of text, sometimes a hundred or more words per knob, far beyond what the balanced rule recommends. It looks wasteful on training day, but amortized across billions of future uses it is the bargain of the century. The student analogy, in its final form: if you are cramming for one exam tomorrow, balance is best; but if you will be answering questions from that same knowledge every day for the rest of your life, it is worth pouring in extra reading now to keep a lean, fast, cheap-to-run mind forever. The right answer depends entirely on whether the big cost is the studying or the lifetime of using what you learned. So the balanced rule is not wrong — it is just answering a narrower question than people assume.\nFigure 3. The balanced rule answers a narrower question than people assume.\nThe sobering footnote: running low on books One quiet assumption underpins this entire story: that there is an endless supply of fresh, high-quality text to read. The whole \u0026ldquo;just feed it more\u0026rdquo; logic works only if there is more genuinely good material to feed. But that is starting to bend. The supply of high-quality human-written text is finite, and the most ambitious models are beginning to bump against its edges. Re-reading the same material does not help much — and past a point it actively hurts, the way cramming the same chapter for the tenth time stops teaching you anything. So the strategy that powers today\u0026rsquo;s best models — small, lean, and enormously well-read — leans on a resource the world is beginning to run short of. The reading list, it turns out, is not infinite.\nThe one-line takeaway We learned to forecast a model\u0026rsquo;s quality before building it; learned the core trade-off is brain-size versus reading; corrected an early bias toward giant, underfed models with the balanced \u0026ldquo;twenty words per knob\u0026rdquo; rule; then realized that for models we will run forever it is smarter to build them small and over-feed them — all while quietly approaching the limits of how much good text there is left to read.\n← Appendix D · Where Words Sit · Contents · Glossary → The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/tutorials/inside-the-machines/appendix-e-chinchilla-scaling/","summary":"\u003ch1 id=\"appendix-e--compute-optimal-scaling-the-chinchilla-story\"\u003eAppendix E — Compute-Optimal Scaling: The Chinchilla Story\u003c/h1\u003e\n\u003cp\u003e\u003ca href=\"/tutorials/inside-the-machines/chapter-09-how-a-model-learns/\"\u003eChapter 9\u003c/a\u003e\n touched on the question of how to split a training budget between a bigger model and more data, and named the Chinchilla correction. This appendix tells that story in full, because it is one of the most illuminating episodes in the field — a case where the conventional wisdom was confidently wrong, got corrected by a single careful study, and then turned out to have a twist that the correction itself missed.\u003c/p\u003e","title":"Compute-Optimal Scaling: The Chinchilla Story"},{"content":"Glossary Every term and acronym used in this book, in plain language. Where a term is the everyday name for an idea explained in a chapter, the chapter is noted.\nActivated parameters — In a mixture-of-experts model, the small slice of the model actually switched on for a single word (the router plus the few chosen experts). Determines speed and running cost. Contrast with total parameters. (Chapter 7)\nAlignment (of a model) — The work of making a model behave the way people want: helpful, honest, well-mannered. Done through the shaping in Chapter 10. (Not to be confused with the translation sense below.)\nAlignment (in translation) — The soft matching, learned automatically by cross-attention, between each output word and the input words it draws from. (Chapter 6)\nAttention — The mechanism by which each word surveys the whole text, decides which other words are relevant to it, and pulls in information from them. The core idea behind nearly every model in this book. (Chapter 3)\nAutoregressive — Generating output one piece at a time, each piece conditioned on those before it. The way language models write. (Chapters 4, 8)\nBackpropagation — The efficient method that computes, in a single backward sweep, which way to nudge every one of a model\u0026rsquo;s knobs to reduce its error. The engine of learning. (Chapter 9)\nBase model — A freshly trained language model: fluent and knowledgeable but not yet shaped into a helpful assistant. (Chapter 10)\nBART — An encoder–decoder model trained by repairing many kinds of deliberately damaged text; strong at summarization. (Chapter 6)\nBERT — The original and most famous encoder (reader) model. (Chapter 5)\nBidirectional — Able to look in both directions at once, at the words before and after. The defining trait of a reader. Opposite of causal. (Chapter 5)\nCausal — Restricted to looking only backward, at what came before. The rule a writer follows so it cannot peek at the answer it is trying to predict. (Chapter 4)\nChinchilla — The study that showed model size and training data should grow in balance — roughly, that earlier giant models had been under-fed with data. (Chapter 9)\nContext window — The amount of text a model can take in and attend to at once. Limited largely by the cost of attention and the growing cache. (Chapters 3, 11)\nContextual representation — The rich, context-aware bundle of numbers an encoder produces for each word; a meaning-fingerprint shaped by the whole sentence. (Chapter 5)\nCross-attention — Attention where the questions come from the writer (the half-finished output) and the answers come from the reader (the memory of the input). The one new mechanism in encoder–decoder models, and the same mechanism that lets a diffusion model follow a text prompt. (Chapters 6, 8)\nDALL·E — A well-known text-to-image diffusion model. (Chapter 8)\nDDIM / DDPM — Two methods for generating images by running a diffusion model\u0026rsquo;s reverse process; DDPM is the original many-step method, DDIM a faster scheme that can skip most steps. (Chapter 8)\nDecoder model — A writer: generates text left to right, one word at a time. GPT, Llama, Claude. (Chapter 4)\nDense model — An ordinary model that uses all of itself for every word. Contrast with mixture-of-experts. (Chapter 7)\nDiffusion model — An image (or audio, video, etc.) generator that learns to predict the noise in a corrupted sample and generates by removing noise step by step, starting from pure static. (Chapter 8)\nDPO — A simpler way to teach a model from human preferences, training directly on preferred-versus-rejected answer pairs without a separate reward model. (Chapter 10)\nEmbedding — The list of numbers that places a token at a location on the \u0026ldquo;map of meaning,\u0026rdquo; where nearness means similarity. Also, a single such vector summarizing a whole sentence or document, used for search. (Chapters 2, 5)\nEncoder model — A reader: takes in text bidirectionally and produces understanding, but cannot generate. The BERT family. (Chapter 5)\nEncoder–decoder model — A translator: a reader bolted to a writer, for turning one text into another. T5, BART. (Chapter 6)\nExpert — In a mixture-of-experts model, one of many copies of the per-word processing stage, each with its own learned settings. (Chapter 7)\nExpert collapse — The failure where the router lazily routes most words to a few favorite experts, leaving the rest as dead weight. Prevented by load balancing. (Chapter 7)\nFine-tuning — Taking a broadly trained model and adapting it, comparatively cheaply, to a specific task or behavior. (Chapters 5, 10)\nGLUE / SuperGLUE — Standard collections of language-understanding tests used to grade reader models. (Chapter 5)\nGPT — A prominent family of decoder (writer) models. (Chapter 4)\nGradient descent — Learning by repeatedly stepping \u0026ldquo;downhill\u0026rdquo; across a landscape of possible knob settings toward lower error. (Chapter 9)\nGuidance (classifier-free guidance) — The control that exaggerates how strongly a diffusion model follows its prompt; the \u0026ldquo;prompt strength\u0026rdquo; slider in image tools. (Chapter 8)\nHead — A small attachment bolted onto a model to turn its internal output into a specific kind of answer (a category, a label per word, a highlighted span). (Chapter 5)\nInference — Using a trained, frozen model to produce outputs. Contrast with training. (\u0026ldquo;Training writes; using reads.\u0026rdquo;) (Chapters 1, 11)\nKV cache — The saved working memory that lets a model avoid re-reading the whole conversation for each new word. Grows with conversation length, and is the main reason long contexts are expensive. (Chapter 11)\nLatent diffusion — Running diffusion in a compressed \u0026ldquo;essence\u0026rdquo; space rather than on full-resolution pixels, which is what made high-quality image generation affordable. (Chapter 8)\nLearning rate — The size of each downhill step during training; the single most consequential training dial. (Chapter 9)\nLlama — A widely used family of openly available decoder models, known for being relatively small and heavily trained. (Chapters 4, 9)\nLoad balancing — The extra pressure during mixture-of-experts training that pushes the router to spread work evenly across experts. (Chapter 7)\nLoRA — A cheap fine-tuning shortcut that freezes the model and learns only a small add-on patch. (Chapter 10)\nLoss — The single number scoring how wrong a model\u0026rsquo;s output was; what training tries to reduce. (Chapter 9)\nMap of meaning — This book\u0026rsquo;s informal name for the high-dimensional space of embeddings, where location encodes meaning. (Chapter 2)\nMasked language modeling — The fill-in-the-blank training game for readers: hide some words and have the model recover them from context. (Chapter 5)\nMixture-of-experts (MoE) — A model design with many experts and a router that activates only a few per word, decoupling total knowledge from per-word cost. Switch, Mixtral, DeepSeek. (Chapter 7)\nMomentum — A refinement to gradient descent that builds up speed along consistent directions and cancels aimless jitter. (Chapter 9)\nNormalization — Plumbing that rescales numbers at each layer to keep a deep model stable. (Chapters 4, 9)\nParameters — The learned numbers — the \u0026ldquo;knobs\u0026rdquo; — that define a model. (\u0026ldquo;Seventy billion parameters\u0026rdquo; counts these.) (Chapter 1)\nPerplexity — A friendly restatement of the loss for language models: roughly, how many words the model is effectively choosing between at each step. (Chapter 9)\nPrefix language model — A middle-ground design using one stack that lets the input portion look around freely while the output portion stays left-to-right. (Chapter 6)\nPretraining — The initial, broad, expensive training of a model on vast data, before any task-specific adaptation. (Chapters 5, 9)\nPrompt — The input you give a model: the text for a language model, or the description for an image model. (Chapters 8, 10)\nReward model — A model trained to predict human preferences, used as an automated stand-in for human taste when shaping an assistant. (Chapter 10)\nRLHF (reinforcement learning from human feedback) — Shaping a model using human comparisons of its answers, usually via a reward model, kept on a leash so it cannot game the score. (Chapter 10)\nRoBERTa — A retrained BERT that showed the original had simply been undertrained; same design, much better results. (Chapter 5)\nRouter (gating network) — The small component in a mixture-of-experts model that decides which experts each word goes to. (Chapter 7)\nSampling — Choosing the next word by rolling dice weighted by the model\u0026rsquo;s odds, rather than always taking the most likely one. (Chapter 11)\nScore (in diffusion) — The direction, at any point, that points toward more realistic images; predicting noise turns out to be equivalent to estimating this. (Chapter 8)\nSelf-attention — Attention in which a sequence asks questions of itself (as opposed to cross-attention). (Chapters 3, 6)\nSequence-to-sequence (seq2seq) — The \u0026ldquo;read one thing, write a different thing\u0026rdquo; task shape, e.g. translation and summarization. (Chapter 6)\nSFT (supervised fine-tuning) — Teaching a base model the shape of being helpful by training it on curated example answers. (Chapter 10)\nStable Diffusion — A widely used, efficient text-to-image diffusion model, made practical by latent diffusion. (Chapter 8)\nT5 — An encoder–decoder model built on the idea that every task is \u0026ldquo;text in, text out.\u0026rdquo; (Chapter 6)\nTemperature — The master creativity dial during generation: low for conservative and predictable, high for adventurous and surprising. (Chapter 11)\nToken — A chunk of text (usually a common word-fragment) — the unit a model actually processes. (Chapter 2)\nTokenization — The fixed, pre-built recipe that chops text into tokens. (Chapter 2)\nTop-k / top-p — Guardrails that discard unlikely options before the model samples a word, keeping only the most plausible handful. (Chapter 11)\nTotal parameters — In a mixture-of-experts model, everything the model stores across all experts — its full capacity. Contrast with activated parameters. (Chapter 7)\nTraining — Building a model by adjusting its knobs on vast data; the only time the model changes. (\u0026ldquo;Training writes; using reads.\u0026rdquo;) (Chapters 1, 9)\nTransformer — The architecture, built around attention and stacked blocks, that underlies essentially every model in this book. (Chapter 3)\nU-Net — The hourglass-shaped network commonly used as a diffusion model\u0026rsquo;s noise-spotter: it squeezes the image to reason globally, then re-expands it, with shortcut wires preserving fine detail. (Chapter 8)\nVariational lower bound — The intimidating-sounding theoretical training goal for diffusion that, once simplified, collapses to \u0026ldquo;predict the noise and minimize the error.\u0026rdquo; (Chapter 8)\n← Appendix E · The Chinchilla Story · Contents ","permalink":"https://www.fanfani.net/tutorials/inside-the-machines/glossary/","summary":"\u003ch1 id=\"glossary\"\u003eGlossary\u003c/h1\u003e\n\u003cp\u003eEvery term and acronym used in this book, in plain language. Where a term is the everyday name for an idea explained in a chapter, the chapter is noted.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eActivated parameters\u003c/strong\u003e — In a mixture-of-experts model, the small slice of the model actually switched on for a single word (the router plus the few chosen experts). Determines speed and running cost. Contrast with \u003cem\u003etotal parameters\u003c/em\u003e. (Chapter 7)\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAlignment (of a model)\u003c/strong\u003e — The work of making a model behave the way people want: helpful, honest, well-mannered. Done through the shaping in Chapter 10. (Not to be confused with the translation sense below.)\u003c/p\u003e","title":"Glossary"},{"content":"There is an old and reliable way to find out what something does: remove it and see what breaks. Surgeons and neuroscientists have done this for centuries — lesion a small region of tissue, watch which ability fades, and infer that the region was responsible for it. The word is ablation, literally a carrying-away, and it has become one of the few tools that gets inside a large language model at all. A modern model is a black box of billions of interacting knobs, and nothing can be read off it directly. So the next best thing gets done instead. A piece is taken away — a layer, an attention head, a training trick, a single direction in the model\u0026rsquo;s internal space — the model runs again, and the damage is measured. The size of the damage is the measure of the part\u0026rsquo;s importance. This piece is about how that removal is defined, how the number is read, and how little a small number proves.\nTwo meanings that share one logic The word \u0026ldquo;ablation\u0026rdquo; covers two related but distinct activities, and both are worth naming before either is used.\nThe first is the ablation study, the workhorse of the research literature. Here the thing removed is a design decision. A model is built with some new component — a normalization scheme, an extra loss term, a different way of encoding position — and a claim is attached to it. A sceptic asks how anyone knows that piece is what helped, rather than something else changed at the same time. The ablation study answers by building an otherwise-identical model with that one component removed or replaced by something plain, training both, and comparing. If the full model scores better, the component earned its place. If the scores match, the component was decorative, which is worth knowing too.\nThe second is mechanistic ablation, the tool of interpretability research. Here the model is already trained and frozen, and the thing removed is not a design choice but a running part of the computation — the output of a particular attention head, the activity of a particular neuron, a specific pattern in the internal state — switched off mid-thought while the model processes one input. The goal is not to judge a design but to reverse-engineer a behaviour: to find which internal components actually carry out a capability the model already has.\nBoth share one logic, borrowed from the lesion studies of biology: establish that a part is responsible for an effect by removing the part and observing that the effect disappears. The difference is grain. One ablates ideas across whole training runs, the other ablates activity inside a single forward pass.\nThe theory: ablation is a causal experiment It is worth being precise about why ablation licenses any conclusion at all, because the failure mode is self-deception. Observing that a component is active when the model does something well does not establish that the component caused the good behaviour — correlation is not causation, and in a densely connected model almost everything correlates with almost everything. Ablation is stronger than observation because it is an intervention. Rather than watching the system run and noting what co-occurs, it reaches in and changes one thing while holding everything else fixed, then measures the consequence. That is the defining move of a controlled experiment.\nFormally, ablation estimates a counterfactual: not what the model does, but what it would have done, on this same input, had one part been absent. Call the model\u0026rsquo;s normal output on input $x$ the quantity $M(x)$, and the output with component $c$ removed $M_{\\neg c}(x)$. The ablation effect is the gap:\n$$\\Delta_c = \\text{score}\\big(M(x)\\big) - \\text{score}\\big(M_{\\neg c}(x)\\big)$$A large positive $\\Delta_c$ means removing $c$ hurt, so $c$ was doing important work. A $\\Delta_c$ near zero means the model barely noticed. That reading is ambiguous: either the part was unimportant, or another part took over its job. The two look identical from here, and separating them takes further experiments rather than a closer look at this number. Telling them apart is the whole discipline.\nWhat can be ablated in a language model The parts that can be carried away span every level of the model, from the coarsest architectural block to the finest thread of internal activity.\nAt the coarsest level sit whole architectural choices — an entire type of layer, the residual shortcut connections, a normalization step, a chunk of the training recipe such as a warm-up schedule or a data-cleaning stage. This is the territory of ablation studies. At a middle level sit structural components of the trained network — a specific layer, one attention head out of the dozens, the entire feed-forward stage of one block, or one expert in a mixture-of-experts model. At the finest level sit units of internal activity — a single neuron\u0026rsquo;s activation, a group of neurons, or a direction: a linear combination of activations that a probe or a sparse dictionary has identified as representing some concept, such as \u0026ldquo;this text is in French\u0026rdquo; or \u0026ldquo;the sentiment is negative\u0026rdquo;.\nAblation also works on the input and data side: remove a category of training data and retrain to see what knowledge vanishes, or remove part of the prompt to see how much the answer depended on it. And it works on information pathways rather than components — cutting the specific route by which one head\u0026rsquo;s output reaches another, while leaving both heads running. That last kind, path patching, traces not just which parts matter but how they talk to each other.\nHow ablation actually works: four ways to remove a part Here the definition does real work, and getting it wrong corrupts everything downstream. In a physical brain, removing tissue is unambiguous. In a neural network, \u0026ldquo;remove\u0026rdquo; has to be defined, because every component outputs some number, and something has to be substituted when it is switched off. The choice matters enormously. There are four common answers.\nThe bluntest is zero ablation: force the component\u0026rsquo;s output to zero, as though it had never fired. Simple, and often misleading. Zero is not a neutral value. It may be a wildly out-of-distribution input the rest of the network has never seen, so the damage measured includes the shock of that unnatural zero and not only the loss of the component\u0026rsquo;s real contribution.\nGentler is mean ablation: replace the output with its average across many inputs. This removes the part\u0026rsquo;s ability to respond to the specific input — it now says the same bland thing regardless of context — while keeping its output in a realistic range. The question it answers is narrower than it looks: how much did it matter that this part could vary.\nMore careful still is resampling ablation, also called noising or interchange ablation: replace the output with the value it produced on a different, randomly chosen input. This destroys the information the component was carrying about this input while keeping every value statistically natural. It is the backbone of activation patching.\nFinally there is the mirror image, activation patching in the positive direction, sometimes called causal tracing or restoration: instead of corrupting a healthy run, start from a corrupted run and paste in the clean activation of a single component, then watch whether that alone restores the correct answer. If restoring one small part flips the model back to correct, that part was carrying the crucial information. Ablation and patching are two ends of the same causal lever. One asks what breaks on removal, the other what heals on restoration.\nFigure 1. Four ways to remove a part, and what each asks.\nHow the damage is measured Once a part is removed, the damage needs a number, and the choice of metric shapes what the number can support.\nThe most general metric is the change in the training loss, or its friendlier cousin perplexity, over a held-out set of text: the drop in probability the wounded model assigns to the correct continuations. For a targeted behaviour, a sharper metric is the logit difference — the gap between the raw score given to the correct answer and the score given to a specific wrong one. Such a gap is a log-odds ratio, so it reads naturally: a difference of one is a factor of about three in the relative odds. Ablating a head that shrinks the gap from a confident $+4.2$ to a hesitant $+0.3$ has caught that head doing most of the work of preferring right over wrong. For distributional questions, the KL divergence between the original output odds and the ablated ones measures how far the whole shape of the prediction moved. For end-task studies the metric can be accuracy on a benchmark, before and after. Whichever you pick, it is fixed before the sweep begins, because a metric chosen after the numbers are in is not a measurement.\nWhere to focus, and the traps that await Ablation is powerful but treacherous, and most of the skill is knowing where to point the suspicion.\nThe deepest trap is redundancy and self-repair. Large models often spread a capability across several components, so removing any one barely moves the metric — not because none matters, but because the survivors compensate. Some models actively self-repair: ablate one head and a downstream component notices the disturbance and adjusts to undo it, hiding the effect. A small $\\Delta_c$ therefore does not establish that a part is useless. The remedy is to ablate groups of related components together rather than one at a time. Then check the story survives.\nA second trap is interaction effects. Two components may each look useless alone and be jointly essential, or one may help only in the presence of another. Single-part ablations miss this entirely. Hence combinations, and hence the drift toward path-based methods that respect how parts cooperate.\nA third is the out-of-distribution shock already named: an ablation that shoves the network into a state it never occupies during normal running produces large damage reflecting the shock, not the lost function. This is why resampling ablation, which keeps every value plausible, is preferred for careful causal claims.\nFinally, in ablation studies of design choices, the classic trap is changing more than one thing at a time. A model that differs from its baseline in three ways at once yields a score difference that says nothing about which of the three was responsible. The entire value of an ablation study rests on removing exactly one thing and holding all else — data, training steps, random seed, everything — fixed.\nFigure 2. Why a small effect does not mean an unimportant part.\nA full worked example: finding the head that decides who received the gift One complete mechanistic ablation, from beginning to end and with concrete numbers, puts the whole method in one place. The task is a simplified version of a classic interpretability probe.\nThe behaviour. A small trained language model is given the sentence \u0026ldquo;When Mary and John went to the shop, John handed a drink to ____\u0026rdquo; and asked for the next word. A competent model answers \u0026ldquo;Mary\u0026rdquo;, not \u0026ldquo;John\u0026rdquo;. The logic a human uses is that John is the giver, having already appeared as the subject of \u0026ldquo;handed\u0026rdquo;, so the other name must be the receiver. The question is which internal parts carry out that piece of reasoning — specifically the part that suppresses the already-used name and promotes the unused one.\nThe metric. The logit difference between the two candidates:\n$$\\text{logit-diff} = \\text{logit}(\\text{\"Mary\"}) - \\text{logit}(\\text{\"John\"})$$On the intact model this comes out at +3.6, a firm preference for the correct name. That number is the thermometer, and every ablation is judged by how far it drops.\nThe suspects. Suppose the model has 12 layers with 12 attention heads each, 144 heads in all. Inspecting all 144 by hand is not a plan. The sweep is systematic.\nThe intervention. Each head in turn receives a resampling ablation: take a corrupted version of the sentence with the names swapped or scrambled, record what that head outputs there, and paste that value in place of the head\u0026rsquo;s real output on the clean sentence. This destroys the sentence-specific information the head was carrying while keeping its output realistic. Then recompute the logit difference. Record the fall.\nThe results. Most heads barely move the number — the logit difference stays near +3.6, which places them outside this task. A handful stand out. Written as the ablation effect $\\Delta = 3.6 - (\\text{logit-diff after ablating})$:\nHead (layer.head) Logit-diff after ablation Effect $\\Delta$ Reading 9.9 +0.4 +3.2 Enormous — carries most of the decision 9.6 +1.5 +2.1 Large — a second major contributor 10.0 +2.0 +1.6 Substantial 10.7 +4.3 −0.7 Negative — this head was working against the answer 3.0 +3.5 +0.1 Negligible (the other 139) ≈ +3.6 ≈ 0 Irrelevant to this task The reading. Three heads — 9.9, 9.6, 10.0 — account for almost the entire preference for \u0026ldquo;Mary\u0026rdquo;: switch them off and the model is nearly a coin-flip between the two names. In the interpretability literature these are name-mover heads, attending from the final position back to the unused name and copying it forward. The negative entry, head 10.7, is as informative. Ablating it raised the logit difference, so it was pushing toward the wrong answer. Such heads exist, often acting as calibrators that keep the confident heads from overshooting, and they are findable only because ablation reports signed effects rather than magnitudes.\nThe confirmation. The sweep is not the end, because of the redundancy trap. Two follow-ups seal the case. First, ablate the three top heads together: the logit difference collapses to roughly −0.2, so as a group they are not merely helpful but necessary — without them the model slightly prefers the wrong name. Second, run the mirror experiment: start from the fully corrupted sentence, where the logit difference sits near zero, and patch back in only head 9.9\u0026rsquo;s clean output. If that single restoration lifts the difference most of the way to +3.6, the head is not only necessary but nearly sufficient to carry the information. When removal breaks the behaviour and restoration repairs it, the causal claim is about as tight as this kind of evidence gets.\nAnd what it does not settle. Out of 144 heads, a findable circuit of about three implements \u0026ldquo;promote the name that was not already used\u0026rdquo;, with at least one other acting as a brake. None of that was visible from watching the model run. But the experiment establishes that those heads are causally involved and nothing more: it does not say what the heads represent, whether the same circuit fires on a sentence with three names, or whether a larger model solves the task this way at all. The article on chain-of-thought faithfulness makes the same point about a different object — causal involvement and description are separate properties, and only the first is what this method returns.\nFigure 3. An ablation sweep over 144 heads.\nWhat to carry forward Ablation is the practice of understanding a model by removing a part and measuring how much worse it gets — a controlled, causal experiment rather than a passive observation. It comes in two forms that share one logic: the ablation study, which removes a design choice across whole training runs to establish that the choice earned its keep, and mechanistic ablation, which switches off a running component mid-computation to reverse-engineer a behaviour. Because removing a part of a network is ambiguous, the definition matters — zeroing, mean-replacing, or most rigorously resampling from other inputs — and so does the metric, whether loss, logit difference or KL divergence. The great pitfalls are redundancy and self-repair, which make important parts look unimportant, and confounds, which let one change masquerade as another; the discipline is removing exactly one thing, testing groups as well as singletons, and confirming that what breaks on removal heals on restoration. The reading it supports is narrower than the effort suggests. A large effect may be the shock of an unnatural input rather than a lost function. A small one proves nothing at all while a survivor can compensate. And even a clean result establishes that a component is causally involved without saying what it represents, which is a narrower finding than the effort suggests — the worked example above pins three heads to one sentence pattern, and says nothing about what those heads do anywhere else.\nThe figures in this article follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/articles/ablation/","summary":"\u003cp\u003eThere is an old and reliable way to find out what something does: remove it and see what breaks. Surgeons and neuroscientists have done this for centuries — lesion a small region of tissue, watch which ability fades, and infer that the region was responsible for it. The word is \u003cem\u003eablation\u003c/em\u003e, literally a carrying-away, and it has become one of the few tools that gets inside a large language model at all. A modern model is a black box of billions of interacting knobs, and nothing can be read off it directly. So the next best thing gets done instead. A piece is taken away — a layer, an attention head, a training trick, a single direction in the model\u0026rsquo;s internal space — the model runs again, and the damage is measured. The size of the damage is the measure of the part\u0026rsquo;s importance. This piece is about how that removal is defined, how the number is read, and how little a small number proves.\u003c/p\u003e","title":"Ablation — Learning What a Model Needs by Taking Pieces Away"},{"content":"Training a neural network is a downhill walk. The knobs start at random, the loss says how wrong the current settings are, and backpropagation hands us, for every knob in the tower, a small number saying which way to nudge it. What it never says is how far. Real training does not take that number at face value; it accumulates momentum and keeps a separate step size for every knob, and that machinery has a name, AdamW, which is the optimizer behind very nearly every large model trained in the last several years. This piece is about the two ideas it combines, and about the single-letter correction — the W — that turned a good optimizer into the default one.\nThe problem with one step size for everything The simplest thing you can do with a gradient is walk against it. Take the number backpropagation gave you, multiply it by some small constant, subtract. One constant, the learning rate, applied to every knob in the model. This works, in the sense that the loss goes down, and for a small model on a clean problem it may be all you need.\nIt stops working for a reason that is easiest to see as terrain. The loss landscape is not a uniform bowl. In some directions it is a narrow ravine with steep walls; in others it is an almost level plain stretching away for miles. A single stride length has to serve both. Pick a stride suited to the plain and you will overshoot the ravine on every step, clattering from one wall to the other and making no progress along the floor. Pick a stride suited to the ravine and the crossing of the plain will take a hundred thousand steps you do not have.\nThe same mismatch shows up if you stop thinking geometrically and just look at the numbers. Consider the embedding row for a rare token — a word that appears in perhaps one batch in a thousand. Nine hundred and ninety-nine times out of a thousand its gradient is exactly zero, and on the thousandth it is enormous. Now consider a normalization gain deep in the tower, which receives a steady, modest gradient on every single step. These two knobs live at completely different scales, and there is no single multiplier that treats both sensibly. Choose one small enough to keep the rare embedding from lurching across the map when its moment finally comes, and the normalization gain will barely have moved by the end of training. Choose one large enough for the gain and the embedding will explode.\nWorse, the right answer drifts. Gradients early in training, when the model is producing noise, are large and pull in roughly the same direction for almost every knob. Late in training they are small and disagree with each other. A step size chosen for the first week is wrong by the third.\nWhat we want is not one stride length but a stride length per knob, set automatically, and revised continuously as training goes on. Adam gets there by keeping two running averages.\nMomentum: remembering which way you were going The first average addresses noise. The gradient we actually receive is not the true gradient of the loss over all the training data — computing that would mean a forward pass over the entire corpus for every step. It is the gradient over one mini-batch, a few hundred thousand tokens sampled more or less at random. That is an honest estimate of the true direction, but a jittery one. Two consecutive batches will disagree substantially about which way any particular knob should go.\nSo rather than trusting the latest reading, keep a running average of the recent ones. This is momentum, and the physical name is apt: a heavy cart pushed by a gusting wind travels in the average direction of the gusts, not in the direction of whichever gust is blowing now. Components of the gradient that point the same way batch after batch accumulate. Components that flip sign at random cancel against themselves and quietly vanish.\nThe ravine benefits twice over. Across the ravine, the gradient alternates in sign as we bounce between the walls, so those contributions annihilate. Along the ravine floor, the gradient is small but always points the same way, so it builds. Momentum turns the shape of the landscape from an obstacle into an ally.\nFigure 1. The ravine, with and without momentum.\nPer-knob pacing: dividing by how rough the ride has been The second average addresses scale, and it is the more interesting of the two. Keep a running average not of the gradient but of the gradient squared. Squaring throws away the sign and leaves only size, so what accumulates is not a direction at all but a rough measure of how violently the ground beneath this particular knob has been shifting lately.\nThen divide the step by the square root of that quantity. A knob whose gradients have been consistently large has been on rough ground, and its step gets scaled down. A knob that has been quiet for a thousand steps has a tiny running average, so when a gradient finally does arrive, the division makes its step correspondingly long.\nThis is the trick that makes a single learning rate workable. After the division, the step is no longer measured in raw gradient units — it is measured in units of this knob\u0026rsquo;s own typical recent gradient. Every knob in the model is put on the same footing, whatever scale it happens to live at, and the learning rate is left to say only how bold we want to be in those normalized units.\nA pinch of math: the four lines that make up Adam Write $g_t$ for the gradient of one knob at step $t$ — the single number backpropagation just handed us. Both running averages are built the same way: mostly the old value, with a splash of the new one.\n$$m_t = \\beta_1 m_{t-1} + (1 - \\beta_1)\\, g_t$$$$v_t = \\beta_2 v_{t-1} + (1 - \\beta_2)\\, g_t^2$$The first, $m_t$, is the smoothed direction. With $\\beta_1 = 0.9$, each new gradient contributes a tenth and the accumulated past keeps nine tenths, so a consistent pull survives and random jitter cancels. The second, $v_t$, is the same recipe applied to the square, and serves as the roughness meter. With $\\beta_2 = 0.999$ it has a far longer memory than $m_t$, which is deliberate: you want your sense of direction to be responsive and your sense of scale to be steady.\nBoth averages start at zero, which makes them badly wrong for the first few hundred steps, since an average that begins at zero is dragged toward zero. Both are therefore inflated back to honest size by dividing out exactly how much of the average is still made of that initial zero.\n$$\\hat{m}_t = \\frac{m_t}{1 - \\beta_1^{\\,t}}, \\qquad \\hat{v}_t = \\frac{v_t}{1 - \\beta_2^{\\,t}}$$At $t = 1$ this multiplies $m$ by ten and $v$ by a thousand; by step ten thousand the correction has faded to nothing. It is bookkeeping rather than a new idea, but without it the opening steps of training are timid in a way that is very hard to diagnose later.\nThe update itself is then one line:\n$$\\theta_t = \\theta_{t-1} - \\eta \\cdot \\frac{\\hat{m}_t}{\\sqrt{\\hat{v}_t} + \\epsilon}$$Read the fraction as direction divided by roughness. The numerator says which way; the denominator says how confidently. The $\\epsilon$, usually around $10^{-8}$, is not a dial but a floor — it exists so that a knob with no recent gradient whatsoever does not divide by something vanishingly small and fly off the map. You do not need to hold all four lines in your head at once; the intuition is that Adam is a smoothed direction, rescaled by a smoothed sense of how much this knob has been moving.\nWeight decay, and the quiet bug that AdamW fixed There is one more ingredient, and it comes from a different tradition entirely. Independently of any gradient, it is useful to pull every weight gently back toward zero on every step — a slack elastic tether tied to the origin, doing nothing when a knob is small, tugging harder the further out it drifts. Smaller weights tend to mean smoother functions and better generalization, and they keep the arithmetic tame. This is weight decay, and it long predates Adam.\nThe traditional way to implement it is to add a penalty on the size of the weights directly to the loss. The optimizer then sees, as part of the gradient, an extra term proportional to the weight itself. With plain gradient descent this is exactly equivalent to shrinking every knob by a fixed fraction each step, which is precisely what we wanted.\nWith Adam it is not equivalent at all, and the departure is worth savouring. That extra term is now part of $g_t$. It flows into $m_t$, it flows into $v_t$, and — this is the damage — it gets divided by $\\sqrt{\\hat{v}_t}$ along with everything else. A knob with large, active gradients has a large roughness meter, so its tether is weakened. A knob sitting quietly with tiny gradients has a small roughness meter, so its tether is amplified. The tidying ends up applied in inverse proportion to how busy each knob is, which is close to the opposite of the intent, and it welds the two dials together: change the learning rate and you have silently changed how much decay you are applying.\nThe fix is disarmingly direct. Do not put the decay in the loss. Apply it as its own separate subtraction, outside the adaptive machinery, after the rescaled gradient step has been computed.\n$$\\theta_t = \\theta_{t-1} - \\eta \\left( \\frac{\\hat{m}_t}{\\sqrt{\\hat{v}_t} + \\epsilon} + \\lambda\\, \\theta_{t-1} \\right)$$That is the whole of the W — decoupled weight decay, from Loshchilov and Hutter, and the only difference between Adam and AdamW. Every knob is now pulled toward zero by the same fraction regardless of its gradient history, the two dials come apart and can be tuned independently, and models trained this way generalize measurably better. It is a one-line change that took several years to notice, which is a fair summary of how a great deal of progress in this field actually happens.\nFigure 3. Coupled decay is applied in inverse proportion to need.\nWhat it costs: three copies of the model Adam\u0026rsquo;s two running averages are not shared. There is an $m$ and a $v$ for every single knob, which means the optimizer carries two extra numbers per parameter, held at full precision because they accumulate over hundreds of thousands of steps and cannot tolerate rounding.\nThe full accounting for a large model trained in mixed precision is sobering. Per parameter: two bytes for the half-precision weight used in the forward pass, two for its gradient, four for the full-precision master copy of the weight, four for $m$, and four for $v$. Sixteen bytes per parameter, of which the weights the model actually consists of are two. A seven-billion-parameter model needs about fourteen gigabytes to hold, and about a hundred and twelve to train.\nThis single fact is why so much of the fine-tuning literature exists. Freeze the base model and you eliminate the master copy, the gradients, and both moments for the overwhelming majority of the knobs — which is precisely the bargain LoRA strikes, and why the frozen base in QLoRA can be squeezed to four bits without touching the small adapter that is still being optimized at full precision. When full training genuinely is required, the usual recourses are to shard the optimizer state across machines so no single device holds all of it, or to keep the moments themselves in eight bits, which turns out to be tolerable in a way that eight-bit gradients are not.\nFigure 2. Sixteen bytes per parameter, of which two are the model.\nThe dials, and what people actually set them to The learning rate remains the one that matters, and it is almost never held constant. Training opens with a warm-up of a few thousand steps ramping from nearly zero, because Adam\u0026rsquo;s roughness meter has not yet seen enough gradients to be trustworthy and a confident step taken on a bad estimate can wreck an initialization . It then decays along a cosine curve to something like a tenth of its peak, so that the model finishes with small careful steps.\nOf the two betas, the first is essentially always nine tenths. The second is often lowered from the classic $0.999$ to $0.95$ for large language models, which shortens the memory of the roughness meter — with a very long memory the optimizer reacts too slowly to a sudden loss spike, and a model that hits one may take thousands of steps to recover. The decay $\\lambda$ is typically a tenth, and is applied to the weight matrices but deliberately not to biases or to normalization gains, which have too few parameters to be worth regularizing and whose natural resting values are not zero anyway. Finally, gradients are usually clipped to a fixed maximum length before the optimizer ever sees them, a seatbelt against the one anomalous batch that would otherwise poison both running averages at once.\nWhat to carry forward AdamW is what the phrase \u0026ldquo;the downhill walk\u0026rdquo; actually means in practice. It keeps two running averages per knob: a smoothed direction, which averages away the noise of mini-batch sampling and lets consistent signals accumulate, and a smoothed roughness, the average squared gradient, which it divides by so that every knob takes a step measured in units of its own recent behaviour rather than in raw gradient units. That division is what allows one learning rate to govern a model whose parameters live at wildly different scales, and it is why Adam works on rare-token embeddings and normalization gains alike. Both averages are bias-corrected in the opening steps, a piece of bookkeeping that matters more than it looks. Bolted on top is decoupled weight decay — a uniform pull of every weight toward zero, applied outside the adaptive rescaling rather than folded into the loss, which keeps the decay from being weakened exactly on the knobs that most need restraining and lets the learning rate and the decay be tuned as independent dials. The price is memory: two full-precision numbers per parameter, which in a full mixed-precision setup means roughly sixteen bytes per knob and an optimizer state several times larger than the model itself. That price is the reason the frozen-base methods exist, and the reason a technique as simple as keeping the moments in eight bits was worth a paper.\nThe figures in this article follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/articles/adamw/","summary":"\u003cp\u003eTraining a neural network is a downhill walk. The knobs start at random, the loss says how wrong the current settings are, and \u003ca href=\"/articles/gradient-descent/\"\u003ebackpropagation\u003c/a\u003e\n hands us, for every knob in the tower, a small number saying which way to nudge it. What it never says is \u003cem\u003ehow far\u003c/em\u003e. Real training does not take that number at face value; it accumulates momentum and keeps a separate step size for every knob, and that machinery has a name, \u003cem\u003eAdamW\u003c/em\u003e, which is the optimizer behind very nearly every large model trained in the last several years. This piece is about the two ideas it combines, and about the single-letter correction — the W — that turned a good optimizer into the default one.\u003c/p\u003e","title":"AdamW — Letting Every Knob Move at Its Own Pace, and Pulling Them All Home"},{"content":"A language model, in its ordinary use, receives an input and produces an output. An agent is what you get when the output is fed back in — when the model acts, observes what happened, and decides what to do next, repeatedly, until it judges the task finished. The change sounds procedural and is not. A single response either succeeds or fails and the user sees which; a loop of thirty steps can fail at step four and spend twenty-six more steps building on the failure, and nobody notices until the end. This piece is about that loop, about the arithmetic of compounding reliability that governs how long it can run, and about the fact that most of the difficulty turns out to be context management rather than reasoning.\nThe loop The structure is small enough to state in a paragraph. The model is given a goal, a set of tools , and whatever context it starts with. It produces some reasoning and an action. The action is executed and its result appended to the conversation. The model is invoked again on the extended conversation, produces the next action, and so on, until it emits something signalling completion or a limit is reached.\nThat is the whole architecture, and the elaborate frameworks built around it are variations on where the reasoning goes, how the history is stored, and what happens when things go wrong.\nWhat makes it powerful is that the model\u0026rsquo;s output becomes its own input, so the system can do things no single forward pass could. It can explore — try something, see it fail, try differently. It can decompose — handle one part, then use the result for the next. It can persist — keep working while a single response would have had to stop. And it gets feedback from the world rather than only from its own predictions, which is the sharpest difference from chain-of-thought reasoning , where the model deliberates but never learns whether any intermediate step was actually right.\nThe arithmetic that governs everything Here is the calculation that explains most agent failures, and it needs no more than multiplication.\nSuppose each step succeeds with probability $p$, independently. Then a task requiring $n$ steps succeeds with probability $p^n$:\n$$P(\\text{success}) = p^{\\,n}$$At $p = 0.95$ — which would be a very reliable step — ten steps give $0.60$, thirty give $0.21$, and fifty give $0.08$. A system whose individual actions are right nineteen times in twenty fails most long tasks. Getting to a useful success rate over thirty steps requires per-step reliability above $0.99$, which is a much harder engineering target than it sounds.\nTwo implications follow, and they shape everything.\nReducing the number of steps is as valuable as improving each one. A tool that accomplishes in one call what previously took five improves the whole system more than a considerable improvement in the model. This is the strongest argument for designing tools around tasks rather than around the underlying API — a single tool that performs a complete operation, rather than six primitives the model must sequence correctly.\nRecovery beats prevention. The independence assumption is what makes the compounding so brutal, and it is also what can be attacked. If a failed step can be detected and retried, its failure no longer terminates the chain. An agent that checks its work, notices an error, and corrects it is not making fewer mistakes — it is breaking the multiplication. Almost every technique that makes agents practical is a form of this: verification, retries, tests, human checkpoints.\nFigure 1. Why nineteen-in-twenty fails most long tasks.\nContext is the real constraint The most consistent surprise for people building these systems is that the binding limit is not the model\u0026rsquo;s intelligence but its context.\nEvery step adds material. The action, the result, the reasoning that produced it. A tool that returns a directory listing, a search result, a file, an error trace — these accumulate, and after twenty steps the conversation may hold a hundred thousand tokens of which the great majority is no longer relevant. The cache grows accordingly, and worse, the dilution and lost-in-the-middle effects mean the original instruction is now buried in the middle of a long transcript, attended to less reliably than anything recent.\nThe practical consequence is that agents drift. They lose track of what they were asked, repeat work they have already done, and pursue subgoals that made sense ten steps ago. This looks like a reasoning failure and is usually an attention failure.\nSo context management becomes the central engineering activity. Summarize the history periodically and replace the raw transcript with the summary. Keep a separate structured record of what has been accomplished and what remains, re-stated at each step rather than reconstructed from the conversation. Write intermediate results to files and hold only references in context. Restate the goal near the end of the prompt where attention is strongest. Give subtasks to fresh invocations with clean context and return only their conclusions, which keeps the parent\u0026rsquo;s history short and is the strongest version of the technique.\nThat last pattern — a coordinating agent delegating to sub-agents whose context it never inherits — is the main structural answer to the problem, and its cost is that the parent cannot see what the child saw, only what it reported.\nFigure 2. What twenty steps leave in the window.\nCommon structures A few arrangements recur often enough to name.\nReason, act, observe. Interleaving explicit reasoning with actions, rather than emitting actions alone. The reasoning is not decoration: it gives the model tokens in which to consider what the last observation implies before committing, and it makes the trace legible to whoever has to debug it.\nPlan then execute. Produce a full plan first, then work through it. This helps on tasks with real structure and hurts on tasks where the right next step depends on what the last one revealed, because a plan made in ignorance is followed past the point where it stopped making sense. The usual compromise is to re-plan periodically rather than once.\nCritique and revise. A second pass — the same model or a different one — examines the output and identifies problems, which are fed back. This works well when the critique can appeal to something external, such as a test suite or a specification, and much less well when the model is judging its own work by its own standards, since the same blind spots produced and then evaluate the answer.\nDelegation. A coordinator decomposes and dispatches to specialists. It is the answer to context accumulation and to genuinely parallel subtasks, and its difficulty is that decomposition is itself a hard judgement, and that a coordinator working from a summary of a summary is several steps removed from the ground truth.\nEvaluation, and why it is the bottleneck Evaluating an agent is substantially harder than evaluating a model, and the difficulty is not incidental.\nThere is often no single correct answer, only outcomes that are more or less acceptable. Cost varies enormously between runs, so quality must be reported against tokens spent rather than alone. The same task run twice takes different paths, so results are noisy and small differences require many trials. And the environments are stateful, so a genuine evaluation must be reset between runs, which makes the whole exercise expensive and slow.\nThe consequence is that agent development is unusually prone to proceeding on impressions. A change feels better on the handful of examples someone tried; whether it is better is frequently unknown. Teams that build reliable systems are almost always the ones that invested early in a real evaluation harness — sandboxed environments, deterministic resets, a suite of tasks with checkable outcomes — and the ones that did not tend to plateau at demonstrations.\nThe honest state of things is that agents work well when the loop is short, the tools are well-designed, the environment gives clear feedback, and success is checkable. They work poorly on long-horizon tasks in environments that do not answer back. That is a narrower claim than the enthusiasm around them suggests, and it is also a description of a rapidly moving boundary — the horizon over which these systems remain coherent has been extending steadily, and the compounding arithmetic says that each increment of per-step reliability buys disproportionately more of it.\nWhat to carry forward An agent is a model whose output is fed back as its own input: it acts, observes, and decides again, in a loop, until the task is done or a limit is hit. The loop\u0026rsquo;s power is that it gets feedback from the world rather than only from its own predictions, which lets it explore, decompose, and recover in ways a single response cannot. Its governing constraint is multiplicative — with per-step reliability $p$ and $n$ steps, success goes as $p^n$, so even a very reliable step fails most long tasks — from which two things follow: reducing the number of steps is worth as much as improving each one, which argues for tools designed around whole tasks rather than primitives; and recovery beats prevention, since detecting and retrying a failed step breaks the multiplication that makes the arithmetic so unforgiving. In practice the binding limit is not reasoning but context: every step accumulates material, the original instruction ends up buried where attention is weakest, and the resulting drift looks like a reasoning failure while being an attention failure — so summarization, external state, and delegating subtasks to fresh invocations with clean context are the central engineering activity rather than a detail. The recurring structures are interleaving reasoning with action, planning versus re-planning, critique loops that work when they can appeal to something external and much less well when a model judges itself, and hierarchical delegation. And evaluation is the real bottleneck, since there is rarely one correct answer, cost varies per run, paths are nondeterministic, and environments are stateful — which is why these systems work well on short loops with checkable outcomes and poorly on long horizons in environments that do not answer back.\nThe figures in this article follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/articles/agents/","summary":"\u003cp\u003eA language model, in its ordinary use, receives an input and produces an output. An agent is what you get when the output is fed back in — when the model acts, observes what happened, and decides what to do next, repeatedly, until it judges the task finished. The change sounds procedural and is not. A single response either succeeds or fails and the user sees which; a loop of thirty steps can fail at step four and spend twenty-six more steps building on the failure, and nobody notices until the end. This piece is about that loop, about the arithmetic of compounding reliability that governs how long it can run, and about the fact that most of the difficulty turns out to be context management rather than reasoning.\u003c/p\u003e","title":"Agents — Turning One Answer Into a Sequence of Actions"},{"content":"The cost of running a model is almost always quoted in operations. Training budgets are priced in floating-point operations, architectures are compared by operations per token, and halving the operation count of a component sounds unambiguously like progress. A great deal of work built on that premise produced no speedup, and some of it produced a slowdown. The arithmetic units on a current accelerator stopped being the scarce resource some time ago; they spend most of their existence waiting for numbers to arrive from memory, far more slowly than they can be consumed. Once that gap is wide enough the cost of a computation is set by how much data it touches rather than how much work it does with it, and a cost model counting only the work will misidentify what is expensive with near-perfect reliability. This piece is about the ratio that decides which of the two limits binds, why the same model sits on opposite sides of it in training and in generation, and what that fact explains about everything built on top.\nComputing got cheap and fetching did not The number worth holding onto is a ratio rather than a speed. On a current datacentre accelerator, the arithmetic units can perform something on the order of a few hundred operations in the time it takes to bring one number in from the large memory pool attached to the chip. Not a few. A few hundred.\nThe gap is no quirk of one design, and it has been widening for decades. Arithmetic is a matter of transistors, which have been getting smaller and more numerous on the familiar schedule; bandwidth is a matter of how many wires run off the edge of a die and how fast each can be driven, a packaging problem that has improved far more slowly. Every generation has added arithmetic faster than it added the means to feed it.\nSo a chip\u0026rsquo;s headline figure, operations per second, is available only to computations that arrive with enough data already in hand. Everything else runs at whatever rate memory can supply, and the headline becomes a claim about what the chip could have done. Nearly every performance surprise further up the stack descends from this ratio.\nIntensity decides which limit binds The quantity that settles which limit applies is arithmetic intensity: operations performed, divided by bytes moved to perform them. It belongs to the algorithm and its implementation, not to the hardware.\nThe hardware supplies its own version of the ratio — peak arithmetic divided by peak bandwidth, the machine balance, which on current parts sits in the region of a few hundred operations per byte. Set the algorithm\u0026rsquo;s intensity beside it, and whichever is smaller names the wall you are standing against. Intensity comfortably above the balance keeps the units fed and the operation is compute-bound, running near the rated speed. Intensity below it starves them and the operation is memory-bound, running at a fraction of that speed which is, near enough, the ratio of the two numbers.\nDrawn out, this is the roofline: attainable performance against intensity, rising in proportion while bandwidth is the constraint, then flattening into a ceiling once arithmetic takes over, with the bend at the machine balance. Where an operation lands tells you what to do about it — below the bend, cut traffic; above it, cut operations. The prescriptions are opposites.\nOperation counts are the most quoted measure of cost and frequently the wrong one. A change that halves the arithmetic and doubles the traffic is a slowdown anywhere left of the bend, and whoever made it will have a table showing an improvement. This is the mechanism behind a long run of refinements that were cheaper on paper and no faster in practice: sparse patterns, factorizations, decompositions trading one dense multiply for several small ones. The multiply had been living above the bend, where its operations were nearly free. The replacements moved data instead.\nFigure 1. The roofline, with training and generation on it.\nTraining is compute-bound and generation is memory-bound, and the same model does both Same weights, same architecture, same chip, two workloads landing on opposite sides of the bend. This is the observation that earns the framework its keep.\nA training step pushes a batch of many sequences, each of many positions, through every weight in the model, so a weight matrix is fetched once and then used thousands of times — once per token in the batch — before it is finished with. Intensity is enormous, the units are saturated, and the run is limited by how many operations the chip can perform. Which is why training budgets are quoted in operations, why that quotation is honest, and why scaling laws are written in units of compute.\nGeneration is the same model in the other regime, and the numbers are startling. To produce one token for one sequence, the model reads every weight out of memory and performs, per weight, one multiply and one add: fourteen gigabytes fetched to carry out roughly fourteen billion multiply-accumulates. An intensity of about one operation per byte, against a machine balance of several hundred — not near the bend but off the bottom of the graph. The units are idle well over ninety-nine percent of the time. They are not computing slowly. They are waiting, and the accelerator is working as an expensive weight-streaming device that multiplies things on the way past.\nTwo things follow. Single-sequence generation speed is a bandwidth specification: divide the model\u0026rsquo;s size in bytes by the accelerator\u0026rsquo;s bandwidth and the time per token falls out, to within a factor that has little to do with the model. And the idle arithmetic is real, paid for, and unspent, on every generation step of every deployment. Most of what follows is people finding ways to spend it.\nA pinch of math: operations per byte, and where the batch enters Write the intensity of a computation as\n$$I = \\frac{\\text{FLOPs}}{\\text{bytes}}$$and the machine\u0026rsquo;s balance, for peak arithmetic rate $\\pi$ and peak bandwidth $\\beta$, as $B = \\pi / \\beta$. Attainable performance is the smaller of the two ceilings,\n$$P = \\min\\!\\left(\\pi,\\; I\\beta\\right)$$which is the roofline in one line: a slope of $\\beta$ until $I$ reaches $B$, and a flat $\\pi$ thereafter.\nNow price the operation that dominates a transformer. Take a batch of $b$ activation vectors of width $d$ against a weight matrix of shape $d \\times d$. What has to move is the weight matrix, $d^2$ numbers; the activations are $bd$ numbers, negligible whenever $b$ is much smaller than $d$, which it is. The arithmetic is one multiply and one add per weight per element of the batch, or $2bd^2$ operations. Counting a byte per weight,\n$$I = \\frac{2bd^2}{d^2} = 2b$$Intensity proportional to batch size, and independent of $d$. The model\u0026rsquo;s size cancels exactly, which explains the two regimes at a stroke: one sequence gives an intensity of two and is hopeless, a hundred sequences give two hundred and reach the bend, and a larger model changes neither verdict. It also locates the threshold, at half the machine balance — which is why serving systems chase batches of a hundred or two rather than of ten, and why little improves once they are past it.\nFigure 2. Intensity is twice the batch, whatever the model size.\nMost of the serving stack is one observation, restated Four techniques usually presented as separate inventions, all of them one argument in different clothes.\nBatching is the business model. Raising the batch size raises intensity in proportion: the weights are read once and used once per sequence, so the expensive half of the work is amortized while the free half multiplies. Throughput climbs almost without cost until intensity reaches the bend, and then stops. What halts it sooner is not arithmetic but space: the attention cache grows with every concurrent sequence and eats the memory the weights were sharing, which is why scheduling at the granularity of one decode step pays. Serving many users at once is not generosity. It is the only regime in which the hardware is being used at all.\nQuantization buys more than its arithmetic saving suggests. Fewer bits per weight means fewer bytes fetched, and bytes are the scarce commodity, so halving a weight\u0026rsquo;s stored width roughly halves the time per generated token, whether or not the chip supports arithmetic at that width. The saving is in the fetch. Which explains an asymmetry: the same change does much less for training, which sits above the bend where traffic was never the problem, and where mixed precision is chosen for the units it unlocks, not the bandwidth it saves.\nSpeculative decoding is the idle arithmetic, spent on purpose. Checking five candidate tokens in one pass costs one streaming of the weights, near enough what generating a single token costs, because the extra arithmetic lands in units that were doing nothing. It works only where the surplus exists, which is why it helps a lone user and quietly stops helping a batched server that has spent the same slack on throughput.\nKernel fusion matters because the cheap operations are not cheap. A chain of elementwise steps — scale, add, activate, normalize — has an intensity near one at every link, and an unfused implementation pays a full round trip to memory per link, reading a tensor and writing it back for a few flops per element; fusing the chain into one pass collapses those trips into one. Such operations are a trivial share of a model\u0026rsquo;s arithmetic and a substantial share of its runtime, and FlashAttention is the ambitious version of the same manoeuvre.\nThe picture is first-order, and wrong in three known ways. It assumes peak bandwidth is reachable, which requires access patterns the memory system likes; a scattered read achieves a fraction of the quoted figure, so the model quietly overpredicts, and the routing traffic of a sparse model is a standing example. It says nothing about latency, so a small operation can be dominated by launch overhead and reach neither limit. And it treats memory as one thing when there is a hierarchy of pools — exactly the distinction that tiling into fast on-chip memory exploits and the roofline cannot see.\nFigure 3. Four serving techniques, one observation restated.\nWhat to carry forward Arithmetic became cheap faster than bandwidth grew, and on a current accelerator the units can perform a few hundred operations in the time one number arrives from memory. Arithmetic intensity — operations per byte moved — set against that machine balance decides which limit binds: above it a computation is compute-bound, running near its rated speed; below it memory-bound, running at roughly the ratio of the two figures, with the roofline drawing a ceiling that rises with intensity and then flattens. For a batched matrix multiply the intensity is about twice the batch size, independent of model size, which explains the central asymmetry. Training pushes thousands of tokens through each weight it fetches, sits well above the bend, and its cost genuinely is the arithmetic. Generating one token for one sequence reads every weight to perform one multiply-accumulate per weight, an intensity of about one, leaving the units idle over ninety-nine percent of the time. From that number descends most of the inference stack: batching amortizes the weight reads until the attention cache rather than the compute runs out of room, quantization speeds generation by moving fewer bytes rather than doing less work, speculative decoding converts idle units into tokens and stops paying once batching has claimed the same slack, and fusing chains of cheap elementwise operations matters because each unfused link pays a full round trip for a few flops. The framework is first-order and misses three things: peak bandwidth needs access patterns the hardware likes, a small operation can be dominated by launch latency instead, and memory is a hierarchy rather than one pool. What it gets right is which question to ask. A change that halves the arithmetic and doubles the traffic is slower, and it will be reported as an improvement.\nThe figures in this article follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/articles/arithmetic-intensity/","summary":"\u003cp\u003eThe cost of running a model is almost always quoted in operations. Training budgets are priced in floating-point operations, architectures are compared by operations per token, and halving the operation count of a component sounds unambiguously like progress. A great deal of work built on that premise produced no speedup, and some of it produced a slowdown. The arithmetic units on a current accelerator stopped being the scarce resource some time ago; they spend most of their existence waiting for numbers to arrive from memory, far more slowly than they can be consumed. Once that gap is wide enough the cost of a computation is set by how much data it touches rather than how much work it does with it, and a cost model counting only the work will misidentify what is expensive with near-perfect reliability. This piece is about the ratio that decides which of the two limits binds, why the same model sits on opposite sides of it in training and in generation, and what that fact explains about everything built on top.\u003c/p\u003e","title":"Arithmetic Intensity — Why Moving Numbers Costs More Than Using Them"},{"content":"Attention is the mechanism everything else in a transformer is arranged around, and it is almost always invoked as though it were already understood — named, granted, and moved past. It repays being paid for properly. The mechanism is not complicated — it is three linear projections, a dot product, a softmax, and a weighted sum, and you could write it on the back of a napkin. What is remarkable is how much follows from it: that a model can resolve a pronoun forty words back, that it can follow an instruction given at the top of a page, that the whole architecture parallelizes well enough to be worth training on a trillion tokens. This piece is about the mechanism itself, the reasoning that produced it, and the two things it conspicuously cannot do on its own.\nThe problem with passing a summary along Before attention, the standard way to process a sequence was to walk along it, maintaining a running summary. Read a word, update the summary, move on. The summary was a fixed-size vector, and by the end of the sentence it was supposed to contain everything worth knowing about the sentence.\nThe trouble is visible as soon as you state it. Everything the model knows about the first word must survive, intact and undamaged, through every subsequent update, squeezed alongside everything it has learned since, in a vector that never grows. A sentence of forty words asks that same fixed budget to hold forty words\u0026rsquo; worth of detail. Information from the distant past does not so much get forgotten as get crowded out.\nThe failure is worst exactly where it matters most. Consider a sentence like the trophy would not fit in the suitcase because it was too large. Resolving it requires comparing two candidates that appeared some distance back, and knowing something about trophies and suitcases. A running summary has to have anticipated that this comparison would be needed and kept the relevant material alive on the off chance. It cannot go back and look.\nThere is also a mechanical objection. A running summary must be computed in order — you cannot update the summary for position ten until you have the summary for position nine. That is a hard constraint on how much of the work can happen at once, and on modern hardware, which is built almost entirely around doing many things simultaneously, it is close to disqualifying.\nAttention resolves both objections with the same move. Rather than compressing the past into a summary and consulting the summary, let every position look directly at every other position and pull out what it needs. Nothing has to be anticipated, because nothing was discarded. And since each position\u0026rsquo;s lookup is independent of the others\u0026rsquo;, they can all happen at once.\nThree roles for every word The question is how a position decides which other positions are worth listening to, and how much of each to take. The answer is that every token plays three roles at once, and each role gets its own vector, produced by multiplying the token\u0026rsquo;s current representation by a learned matrix.\nPicture a noticeboard. Each token pins up a small advertisement describing what it has to offer — that it is a plural noun, that it is the subject of the sentence, that it concerns cooking. That advertisement is the token\u0026rsquo;s key. Each token also brings the goods themselves, the content it will hand over to anyone who takes it up: this is its value. And when a token\u0026rsquo;s turn comes to gather information, it posts a want-ad describing what it is looking for — a preceding noun, an unresolved referent, the object of this verb. That is its query.\nThe mechanism is then exactly what the picture suggests. Compare a token\u0026rsquo;s query against every key on the board. Where a query and a key match well, that token\u0026rsquo;s value is taken in large measure; where they match poorly, it is taken in small measure or not at all. What arrives back is a blend of values, weighted by how well each advertisement answered the request.\nThree points about this are worth making explicit, because each is doing real work.\nThe comparison is a dot product, which is to say a measure of alignment between two directions — the same operation, and the same geometric reading, as the one that turns a final state into logits . Nothing is being matched symbolically; a query and a key \u0026ldquo;match\u0026rdquo; when they point the same way in a space the model arranged for itself.\nThe three roles are genuinely separate. What a token advertises need not resemble what it hands over. A token can announce \u0026ldquo;I am a date\u0026rdquo; while the value it supplies is the date itself; the advertisement is for finding, the value is for using, and separating them lets the model be found on one basis and useful on another.\nAnd the query is asked from the token\u0026rsquo;s current state, not from the raw word. By the time a token reaches the tenth block, its representation already encodes a great deal of context, so the question it asks at layer ten is a far more sophisticated question than the one it asked at layer one. Attention is applied repeatedly, and each application asks better questions than the last.\nFigure 1. Query, key and value on a noticeboard.\nA pinch of math: scores, scaling, and the mask Stack the queries for all positions into a matrix $Q$, the keys into $K$, and the values into $V$. Then the whole mechanism is one line:\n$$\\text{Attention}(Q, K, V) = \\text{softmax}\\!\\left(\\frac{QK^\\top}{\\sqrt{d_k}} + M\\right) V$$Take it piece by piece. $QK^\\top$ is every query dotted against every key, producing a square grid of scores with one row per asking position and one column per answering position. The softmax turns each row into a set of weights that sum to one, so every position distributes a fixed total of attention across the candidates available to it. Multiplying by $V$ then takes the corresponding weighted blend of values.\nTwo details in that line deserve explanation, because they are the parts nobody guesses.\nThe division by $\\sqrt{d_k}$, where $d_k$ is the width of a key, is a variance correction. A dot product of two vectors of length $d_k$ is a sum of $d_k$ terms, and the spread of such a sum grows with $\\sqrt{d_k}$. Without the correction, wider keys produce systematically larger scores, the softmax receives inputs spread over a large range, and it saturates — one weight goes to essentially one and the rest to essentially zero, which both destroys the blending and flattens the gradient to nothing. Dividing by $\\sqrt{d_k}$ keeps the scores in a sane range regardless of width. It is a small thing that the architecture does not work without.\nThe $M$ is the mask, and it is how the same equation serves different purposes. It is a grid added to the scores before the softmax, containing zero where attention is permitted and negative infinity where it is not — and negative infinity, exponentiated, is exactly zero, so a forbidden connection receives no weight at all rather than merely a small one. For a model that writes, the mask forbids every position from looking forward, which is what makes it legitimate to train on all positions of a sequence simultaneously: each position is predicting its successor without having seen it. For a model that reads, the mask is empty and every position sees every other. The difference between an architecture that generates and one that comprehends is, at this level, one triangular grid of negative infinities.\nFigure 2. One triangular grid of negative infinities.\nMany heads, many conversations at once One set of queries, keys, and values would force the model to express every kind of relationship through a single comparison. A pronoun\u0026rsquo;s referent, a verb\u0026rsquo;s subject, a quoted phrase\u0026rsquo;s opening mark, the topic of the paragraph — all of these would have to be found by the same yardstick, and a query that is good at one is unlikely to be good at another.\nSo the mechanism is run several times in parallel, on different slices of the representation. The vector is divided into equal parts, each part gets its own query, key, and value projections, each part performs the whole procedure independently, and the results are concatenated and passed through one final matrix that mixes them back together. These parallel copies are the heads.\nCrucially the total work is unchanged: sixteen heads each a sixteenth as wide cost what one full-width head would cost. What is bought is not capacity but independence — sixteen different notions of relevance, applied simultaneously, each free to specialize. And they do specialize, without instruction. Examine a trained model and you will find heads that attend almost entirely to the immediately preceding token, heads that track matching brackets, heads that carry a subject forward across clauses, heads that appear to do nothing identifiable at all. Nobody assigned these jobs. They are what the division of labour settled into.\nThe price: everything looks at everything The elegance has a bill attached, and it is the defining constraint on the whole architecture.\nThat grid of scores has one entry for every pair of positions. A sequence of length $n$ therefore requires $n^2$ comparisons, per head, per layer. Doubling the length of the input quadruples the attention work. At a thousand tokens this is unremarkable — the rest of the network dominates. At a hundred thousand tokens it is the entire cost of the model, and the score grid alone, if it were ever written down in full, would occupy more memory than the model\u0026rsquo;s weights.\nNearly everything that follows in the practical literature is a response to this one fact: computing attention without materializing the grid , restricting which positions may see which , sharing keys and values between heads so that fewer need storing, or replacing the mechanism with something cheaper that behaves approximately the same. The quadratic term is the reason there is a field of research here at all rather than a solved problem.\nFigure 3. Where the quadratic term takes over.\nTwo things attention does not do It is worth ending on the mechanism\u0026rsquo;s honest omissions, because both are supplied elsewhere and both are easy to assume are included.\nThe first is order. Look again at the equation and notice that nothing in it refers to position. Attention compares every query against every key on equal terms, and if the tokens of a sentence were shuffled the set of comparisons would be identical. Attention is entirely blind to word order — a startling property for a mechanism used to model language — and position must be injected separately , most often by rotary embeddings .\nThe second is memory of its own work. Each application of attention consults the keys and values available to it and produces an output; it has no notion that these keys and values might be worth keeping. During training that does not matter, since the whole sequence is present at once. During generation it matters enormously, and the machinery for retaining them is the KV cache — which, note, is a cache of exactly two of the three roles, for reasons that the separation described above makes obvious.\nWhat to carry forward Attention replaces a running summary with direct access: rather than compressing the past into a fixed-size state and hoping the right things survived, every position reaches back and takes what it needs from every other, which removes the bottleneck and, because the lookups are independent, allows the whole sequence to be processed at once. Each token plays three roles, each a learned projection of its current state — a query saying what it is looking for, a key advertising what it offers, and a value carrying what it hands over — and the separation of key from value is what lets a token be found on one basis and be useful on another. Scores are dot products between queries and keys, divided by the square root of the key width to stop the softmax saturating as the model gets wider, and shifted by a mask of negative infinities that decides who may see whom: triangular for a model that writes, empty for one that reads. The procedure is run in parallel across several heads on slices of the representation, which costs nothing extra and buys independent notions of relevance that specialize on their own. The price is that every pair of positions must be compared, so cost grows with the square of the length — the single constraint from which most of the practical engineering around transformers descends. And two things are not included: attention is blind to word order, which must be supplied separately, and it has no memory of the keys and values it produced, which must be cached explicitly when generating one token at a time.\nThe figures in this article follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/articles/attention/","summary":"\u003cp\u003eAttention is the mechanism everything else in a transformer is arranged around, and it is almost always invoked as though it were already understood — named, granted, and moved past. It repays being paid for properly. The mechanism is not complicated — it is three linear projections, a dot product, a softmax, and a weighted sum, and you could write it on the back of a napkin. What is remarkable is how much follows from it: that a model can resolve a pronoun forty words back, that it can follow an instruction given at the top of a page, that the whole architecture parallelizes well enough to be worth training on a trillion tokens. This piece is about the mechanism itself, the reasoning that produced it, and the two things it conspicuously cannot do on its own.\u003c/p\u003e","title":"Attention — Letting Every Word Ask the Others What It Needs"},{"content":"The gradient the model actually receives is never the gradient of the thing we want to minimize. What we want is the average loss over all text; what we compute is the average loss over a few hundred thousand tokens drawn at random. The second is an estimate of the first, and how many tokens go into it is a choice — the batch size. Larger batches give a cleaner estimate and fewer steps per unit of data; smaller batches give a noisier estimate and more steps. It sounds like a straightforward accuracy-versus-cost trade, and it is not, because the noise turns out to be useful, because the returns from reducing it stop abruptly at a threshold that can be calculated, and because on the hardware these models run on the choice is entangled with how many machines you are using. This piece is about that threshold and what sits on either side of it.\nTwo things the batch size controls Separate the two effects at the outset, because conflating them causes most of the confusion.\nThe first is statistical. Averaging over more samples reduces the variance of the estimate, and it does so in the familiar way: the standard error of a mean over $B$ samples falls as $1/\\sqrt{B}$. Quadrupling the batch halves the noise. This is a decelerating return, and the deceleration matters.\nThe second is mechanical. Every optimizer step costs a fixed overhead — a synchronization across all the machines involved, a gathering of gradients, an update applied to every parameter, a set of communications that do not depend on how much data went into them. Fewer, larger steps amortize that overhead better. On a cluster of thousands of accelerators, the fixed cost of a step is substantial, and there is real pressure to make steps as large as possible so there are fewer of them.\nThese pull in the same direction up to a point, which is why large-scale training uses batches of millions of tokens. What stops them is that the statistical benefit runs out while the cost does not.\nThe noise is not simply a defect Before the threshold, a word about what the noise is doing, because the instinct that a cleaner gradient must be better is wrong.\nGradient descent with a noisy gradient does not walk straight downhill; it jitters. That jitter has consequences that are difficult to characterize precisely but are consistently observed. It shakes the parameters out of narrow crevices in the loss landscape and leaves them in wider basins, and models sitting in wide basins tend to generalize better, plausibly because a wide basin means the loss is insensitive to small perturbations of the weights and therefore, loosely, to small differences between the training data and the world. Very large batches, trained naively, are known to find sharper solutions and to generalize slightly worse than smaller ones at equal data — the \u0026ldquo;generalization gap\u0026rdquo; that appeared when people first tried to train with enormous batches to exploit large clusters.\nMost of that gap turned out to be fixable by adjusting the learning rate rather than being intrinsic. But the underlying point stands: noise in the gradient is not purely a defect to be minimized. It is a source of implicit regularization, and a training setup that removes it entirely has to replace it with something.\nA pinch of math: the critical batch size The threshold can be derived, and the derivation is the most useful thing here.\nThink of the gradient estimate as the true gradient plus noise. The true gradient is what makes progress; the noise contributes nothing on average and merely wastes some of each step. When the batch is small, the noise dominates and averaging more samples buys a great deal. When the batch is large, the estimate is already close to the true gradient and averaging more buys almost nothing — you are paying twice the compute to remove noise that was no longer in the way.\nThe crossover is where the noise contributes about as much as the signal, and it is captured by a single quantity with units of samples:\n$$B_{\\text{crit}} = \\frac{\\text{tr}(\\Sigma)}{\\|G\\|^2}$$where $G$ is the true gradient and $\\Sigma$ is the covariance of the per-sample gradients — the numerator measuring how much individual examples disagree with each other, the denominator measuring how strong the consensus direction is. This is the gradient noise scale, and it says something intuitive: the right batch size is set by the ratio of disagreement to agreement among your examples.\nIts consequence is a clean statement about diminishing returns. Write $S$ for the number of optimizer steps needed to reach some target loss and $E$ for the total examples processed. Then to a good approximation\n$$\\frac{S}{S_{\\min}} \\;=\\; 1 + \\frac{B_{\\text{crit}}}{B}, \\qquad\\qquad \\frac{E}{E_{\\min}} \\;=\\; 1 + \\frac{B}{B_{\\text{crit}}}$$Read the two together. Below the critical size, doubling the batch nearly halves the number of steps at almost no extra data — steps are cheap to buy. Above it, doubling the batch barely reduces the step count while doubling the data consumed — you are paying in compute for time you no longer save. At exactly $B_{\\text{crit}}$ you sit at the corner, using twice the minimum steps and twice the minimum data, which is the balanced point.\nSo the batch size is not a quality dial at all. It is a dial between wall-clock time and total compute. Below critical, you are wasting time; above it, you are wasting compute. Where you want to sit depends on whether you are short of hardware or short of patience, and organizations training frontier models are generally short of patience and run above critical deliberately.\nThe noise scale is not a constant. It grows as training proceeds — early on, every example agrees that the model is badly wrong and the consensus is overwhelming, so a small batch suffices; later, the remaining disagreements between examples are what is left to learn from, and a larger batch is needed to hear the signal over them. It grows with model capability for the same reason, and it varies by domain: reinforcement learning has a noise scale orders of magnitude above supervised learning, which is why RL runs use batches that look absurd next to pretraining. This is the honest explanation for a practice that otherwise looks superstitious — ramping the batch size up over the course of a training run, keeping it near critical as critical moves.\nFigure 1. Steps and data against the critical batch.\nCoupling to the learning rate Change the batch size and you must change the learning rate, or you have changed two things at once.\nThe rule for plain gradient descent is linear: double the batch, double the learning rate, so that the total distance travelled per example stays the same. The rule for a noise-dominated regime is the square root, since that is how the noise falls. In practice, with AdamW , the square-root rule matches observation better across most of the useful range, and the linear rule is the right instinct only when batches are small.\nBoth rules break at the top. Beyond some learning rate the training becomes unstable regardless of how much data went into each gradient, because the step is simply too large for the curvature of the landscape. This is what caps batch size in practice: past a point, the batch is large enough to support a bigger step but the landscape is not, so the extra data buys nothing at all. The standard mitigations are the warm-up described under AdamW , which lets the optimizer\u0026rsquo;s running estimates settle before large steps are taken, and gradient clipping, which bounds the damage from an anomalous batch.\nGetting a large batch out of small hardware One practical matter, since the numbers above are unreachable on any single device.\nA batch of four million tokens does not fit in the memory of one accelerator, and often not in a thousand. Two mechanisms bridge the gap. Gradient accumulation runs several small forward and backward passes in sequence, adding their gradients into a buffer, and applies one optimizer step at the end — mathematically identical to a single large batch, and trading time for memory. Data parallelism runs the small passes simultaneously on different devices and averages the gradients across them before the step, which is the same arithmetic executed concurrently at the cost of a communication.\nBoth produce the same update. What differs is the resource spent, and real training runs use both at once: a per-device micro-batch chosen to fill the memory available, multiplied by an accumulation count, multiplied by the number of data-parallel replicas, to reach the target. The consequence worth noting is that the batch size a training run reports is a logical quantity assembled from three separate settings, and that changing the hardware layout without adjusting the others silently changes the optimization.\nFigure 2. A logical batch assembled from three settings.\nWhat to carry forward The batch size sets how many examples are averaged into each gradient estimate, and it does two separate things: it reduces the estimate\u0026rsquo;s noise, which improves as the square root of the batch and therefore decelerates, and it amortizes the fixed overhead of a step, which does not decelerate. Because the statistical return runs out while the mechanical one continues, there is a threshold — the critical batch size, given by the ratio of how much individual examples disagree to how strong their consensus is. Below it, doubling the batch nearly halves the number of steps for almost no extra data; above it, doubling the batch consumes twice the data while barely reducing the steps. So this is not a quality dial but a trade between wall-clock time and total compute, and frontier runs sit deliberately above critical because they are short of time rather than short of hardware. The threshold moves during training, rising as the easy consensus is exhausted and only the disagreements remain, which is the real justification for ramping batch size up over a run, and it is far higher in reinforcement learning than in pretraining. Changing the batch requires changing the learning rate with it — roughly as the square root in the regime that matters — up to a ceiling imposed by the landscape rather than by the data, beyond which no amount of extra averaging permits a larger step. And since no device holds millions of tokens, the reported batch is a logical quantity assembled from a per-device micro-batch, an accumulation count, and a replica count, all three of which must move together to keep the optimization unchanged.\nThe figures in this article follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/articles/batch-size/","summary":"\u003cp\u003eThe gradient the model actually receives is never the gradient of the thing we want to minimize. What we want is the average loss over all text; what we compute is the average loss over a few hundred thousand tokens drawn at random. The second is an estimate of the first, and how many tokens go into it is a choice — the \u003cem\u003ebatch size\u003c/em\u003e. Larger batches give a cleaner estimate and fewer steps per unit of data; smaller batches give a noisier estimate and more steps. It sounds like a straightforward accuracy-versus-cost trade, and it is not, because the noise turns out to be useful, because the returns from reducing it stop abruptly at a threshold that can be calculated, and because on the hardware these models run on the choice is entangled with how many machines you are using. This piece is about that threshold and what sits on either side of it.\u003c/p\u003e","title":"Batch Size — How Much Evidence to Gather Before Each Step"},{"content":"A reasoning model shows its working, and the working is the most reassuring thing about it. Ask something hard and there it is on the page: an approach considered, a difficulty noticed, a first attempt discarded, a conclusion reached. It reads like an account of how the answer was arrived at, and it is hard not to read it that way: the format is borrowed wholesale from how people explain themselves, and nobody has practice at reading fluent prose as anything other than a report. Then someone runs the obvious experiment. Slip a hint into the prompt, watch the answer move to match it, and read the accompanying reasoning: a clean, sensible argument that never mentions the hint. Nothing malfunctioned. The model did what it was trained to do. This piece is about the difference between reasoning that does work and reasoning that describes work, why the second was never trained for, and what that leaves of the case for reading traces at all.\nTwo claims travel under one name The phrase chain-of-thought faithfulness covers two quite different claims, and most of the confusion here comes from letting them share a name. Separating them is the one genuinely useful move available, so it is worth making first.\nThe first claim is that the written steps are a computational aid. A model emits one token at a time, each from a single pass through its layers, and carries nothing between passes except the tokens already on the page. On that reading the trace is working storage: intermediate results are written down because there is nowhere else to put them, and the next pass reads them back. The causation runs forward, from tokens to answer.\nThe second claim is that the written steps are an explanation — that the text reports why the answer came out as it did, so reading it tells you what produced the conclusion. Here the causation runs backward, from an answer to an account of it. This is the claim in play whenever someone says a model\u0026rsquo;s reasoning can be audited, checked, or watched for intent.\nThe two are independent, and the first does not deliver the second. Consider long multiplication done on paper. The small digits carried above each column are load-bearing; rub them out and the product comes out wrong. They are also not an account of anything, and would tell a reader nothing about why the algorithm works. Being necessary to a computation and being a description of it are separate properties. The first is well supported for language models. The second is the one in question.\nFigure 1. A computational aid is not an explanation.\nRemoving the trace degrades the answer The forward claim rests on an architectural fact, not on interpretation. A model spends a fixed amount of computation per token — the same tower of layers whether the question is trivial or brutal — and keeps no scratch memory once a token is emitted, so whatever it worked out during a pass is discarded unless written into the context. The context is the only place intermediate state can live, and writing more of it is the only way to spend more computation on a harder problem. This is the mechanism behind reasoning models , and behind buying accuracy with more computation at inference time.\nThe test that settles it is causal rather than observational, and it is the same logic as any ablation : take the part away and measure the damage. Deny a model its intermediate tokens and require the answer immediately, and performance on multi-step problems falls, sometimes to the floor. Truncate the trace partway and accuracy degrades in proportion. Corrupt an intermediate result and later steps inherit the corruption. The tokens are doing work, and removing them measures it.\nOne awkward result belongs here rather than tidied away. On some tasks, replacing the reasoning with meaningless filler — repeated punctuation, padding tokens with no content — recovers part of the benefit, because what the extra positions supply is forward passes and somewhere to put activations, not necessarily arguments. The finding is uneven and should not be overstated, but it points at the right thing. The ablation establishes that the tokens are causally involved. It says nothing about whether their content describes what they enable.\nA pinch of math: what writing intermediate tokens buys The accounting is simple, and worth doing precisely, because it shows what the forward claim gives and where it stops.\nA transformer with $L$ layers computes a fixed function from context to a distribution over the next token. Whatever serial dependency chain it can carry out for one token — this depends on that, which depends on that — is bounded by the number of layers, so the serial depth available for a single answer token is on the order of $L$. That bound does not move with the difficulty of the question.\nNow let the model write $m$ tokens before answering. Each new token is computed from a context that includes every earlier one, so the result of pass $k$ is available as input to pass $k+1$. The chains compose, and the reachable serial depth becomes\n$$\\text{depth} \\;\\sim\\; mL$$which is the whole of the computational case. A trace of a thousand tokens turns a fixed budget of a few dozen sequential steps into tens of thousands. That is not a rhetorical gain but a different class of computation, bought at the price of generating text.\nThen notice what the same accounting does not contain. The object computed is $p(y \\mid x, t_1 \\ldots t_m)$, the distribution over the answer given the prompt and the intermediate tokens. Nothing in it requires $t_1 \\ldots t_m$ to describe the function they are fed into. They need only condition the later passes usefully. True, relevant, well-formed sentences will do that. So will a sequence merely correlated with the right continuation. So, on some tasks, will padding.\nFigure 2. Serial depth: L against mL.\nThe trace omits the thing that moved the answer The experiment that put this on the agenda is unglamorous. Introduce something into the prompt that demonstrably changes the answer — a suggestion that one option is correct, a hint about which choice someone else picked, an ordering of options, a cue about who is asking — confirm by comparing runs that the answer moved because of it, then read what the model wrote. Frequently the trace does not mention the influence at all. It builds a coherent, step-by-step argument that terminates at the answer the influence produced.\nWhat this shows is narrower than the headline. It shows the trace is not a reliable causal account of what determined the output, which sinks the explanatory reading as a guarantee. It does not show the trace is unrelated to the computation; the same models lose accuracy when it is removed, so the tokens remain load-bearing. Both results hold at once. The word rationalization is tempting and carries an implication that does not apply: there is no motive, no awareness of a discrepancy, no concealment. There is a conditional distribution over continuations, shifted by something for which no reporting channel exists.\nNothing in training ever asked for accurate self-report. That makes the finding expected rather than surprising. Pretraining rewarded the trace for resembling human reasoning in a corpus. Preference training rewarded it for being preferred by a rater or by a reward model standing in for one, which selects for traces that look sound. Training with verifiable rewards attaches the signal to the final answer and says nothing about the tokens preceding it, so intermediate text is shaped only by whatever raises the chance of a correct ending. Accuracy of description was in none of these objectives. The corollary is uncomfortable: optimizing harder on outcomes gives the trace more freedom to diverge, not less, since the only constraint on it is that it help.\nAnd the sentence is not the unit. A fluent paragraph has no reason to track the mechanism, because the mechanism is spread across directions and components that do not correspond to sentences at all — the arrangement described by the study of superposition , where features occupy overlapping directions and one component takes part in many unrelated computations. Asking natural language to describe that is asking one representational scheme to summarize another it was never fitted to.\nWhat this costs in practice The consequences are asymmetric, and the asymmetry is the whole case for reading traces at all.\nMonitoring is weakened but not worthless. If serial computation must pass through written tokens, a model working toward something unwanted must do some of that working on the page. So a trace that reveals an unwanted intention is meaningful evidence, while a clean trace is weak evidence of a clean process. The directions are not symmetric: presence is informative, absence nearly uninformative. That, rather than any claim about honesty, is the basis for treating traces as a safety signal .\nTraining against the trace erodes it. Optimize the visible reasoning to look acceptable and the pressure falls on visibility, not on the underlying computation, which relocates into parts of the network no one is reading. The trace becomes less informative in precisely the measure that it becomes more presentable. Keeping the monitoring value means accepting traces that are sometimes ugly.\nProcess supervision inherits the problem. Rewarding good-looking intermediate steps rewards the appearance of good steps, since that is what the grader can see — the same difficulty as any learned judge, arriving in a new place, and it applies equally when the grader is another model providing feedback .\nAnd the measurement is thin. Faithfulness is assessed by intervention rather than inspection, because inspection cannot distinguish an accurate report from a plausible one. The interventions are indirect, results vary by model, by task, and by how the influence is introduced, and there is no agreed metric — which is why confident findings point in opposite directions, one set reading traces as near-useless, another noting the failures cluster in particular kinds of manipulated prompt. The disagreement is less about the data than about what quantity is being estimated. Practice is well ahead of the measurement here.\nFigure 3. Presence is informative, absence nearly is not.\nWhat to carry forward Two claims share the name faithfulness, and keeping them apart is most of the work. The written trace is a computational aid: a model spends a fixed amount of computation per token and keeps no scratch memory between passes, so the context is the only place intermediate state can live, and writing tokens is the only way to buy more sequential computation for a hard problem — which lifts the reachable serial depth from roughly the layer count to that count times the number of tokens written. It is supported causally: removing, truncating, or corrupting the trace measurably damages performance on multi-step problems. The second claim, that the trace explains the answer, does not follow from the first and does not hold in general. Introduce something into a prompt that provably moves the answer, and the reasoning frequently omits it while constructing a sound-looking argument for the answer the influence produced. This is not concealment and not a malfunction; nothing in pretraining, preference training, or outcome-based reward scored the trace for describing the computation it enables, only for helping produce a good ending, and the mechanism in question is distributed across components no sentence corresponds to. Optimizing harder on outcomes loosens the coupling rather than tightening it. What survives is an asymmetry: because serial reasoning must pass through visible tokens, a trace showing something unwanted is real evidence, while a clean trace is weak evidence of a clean process — and that asymmetry degrades the moment the trace is trained to look acceptable, which pushes computation out of view without removing it. Faithfulness can only be probed by intervention, indirectly, and the results do not agree across models or tasks. Traces are being relied on well in advance of anyone being able to say how much they can bear.\nThe figures in this article follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/articles/cot-faithfulness/","summary":"\u003cp\u003eA reasoning model shows its working, and the working is the most reassuring thing about it. Ask something hard and there it is on the page: an approach considered, a difficulty noticed, a first attempt discarded, a conclusion reached. It reads like an account of how the answer was arrived at, and it is hard not to read it that way: the format is borrowed wholesale from how people explain themselves, and nobody has practice at reading fluent prose as anything other than a report. Then someone runs the obvious experiment. Slip a hint into the prompt, watch the answer move to match it, and read the accompanying reasoning: a clean, sensible argument that never mentions the hint. Nothing malfunctioned. The model did what it was trained to do. This piece is about the difference between reasoning that does work and reasoning that describes work, why the second was never trained for, and what that leaves of the case for reading traces at all.\u003c/p\u003e","title":"Chain-of-Thought Faithfulness — Whether the Reasoning Shown Is the Reasoning Done"},{"content":"A team builds an agent that works beautifully for six steps and then begins repeating itself, and the diagnosis they reach for is that the model is not clever enough. They try a larger one. The repetition continues, because the problem was never the reasoning: by step twelve the original instruction sits forty thousand tokens back, sharing a window with three stale directory listings and the full text of a file the agent read once for a reason it can no longer recover. The name that has settled on the business of managing this is context engineering, which sounds like prompt writing with a grander vocabulary and is not quite that. What distinguishes it is that the decisions have measurable costs and trade against one another, which is what makes something an engineering problem rather than a set of habits. This piece is about the budget every component competes for, the ordering that the cache and the attention pattern argue about separately, and the losses that compaction and tool output quietly impose.\nThe window is a budget, and it is shared Three facts, none of them remarkable alone, combine into a constraint worth stating carefully.\nThe window is fixed. Whatever the advertised number, it is a hard ceiling measured in tokens, and it cannot be negotiated upward at request time by any amount of care in how the request is phrased.\nEverything drawing on it draws on the same allowance. The system instruction, the tool definitions, the retrieved documents, the conversation so far and the thing the user has just asked all occupy the same space, and nothing distinguishes them at the level the limit applies to.\nAttention inside the window is not uniform. A token that is present is not thereby a token that is used, since the model distributes a fixed total of attention weight across every position available and material that has to compete against a hundred thousand rivals fares worse than the same material competing against a thousand.\nAny one of these would be a nuisance to work around. Together they mean the interesting decisions are allocations rather than improvements, and an allocation cannot be optimized without knowing what it is being taken from. This is also why the problem does not dissolve when windows grow. A larger budget changes the numbers and leaves the structure intact, and the third fact gets worse rather than better as the first relaxes.\nFigure 1. Five components, one allowance.\nOrder gets decided twice, by the cache and by attention A working context for anything non-trivial holds five kinds of material at once, and every token given to one is a token unavailable to the rest. The system instruction establishes the behaviour wanted. The tool definitions describe what can be called, and with fifty tools they are not a footnote but a substantial block of prose that is re-sent every time. The retrieved documents supply the facts. The history carries what has happened. The current request is the only part that cannot be trimmed, since without it there is nothing to answer.\nThe cost is not only the window. It is also the money and the wait, because the whole context is read again on every turn — the model has no memory between calls, so a conversation is not continued so much as replayed with one more line. What makes the replaying affordable is the KV cache , which stores the per-token intermediate results so that a prefix already processed does not have to be processed twice. The consequence for ordering is direct and slightly unglamorous. A prefix that is byte-identical across turns can be reused; an edit anywhere invalidates the cache from that point onward, because every later token\u0026rsquo;s representation depended on what came before it. Inserting the current date at the top of a system instruction throws away the cached form of everything beneath it.\nThat is the strongest practical argument for putting the invariant material first — instruction, then tools, then the slowly changing material, then the volatile — and it is worth being clear that it is a systems argument about cache reuse rather than a claim about where the model reads best. The two arguments happen to agree here. They do not always.\nPosition matters, and unevenly. Models attend less reliably to the middle of a long input than to either end, a pattern consistent enough across long contexts to have acquired the name lost in the middle. For ordering retrieved passages this has an awkward implication: sorting passages by relevance and pasting them in order puts the fourth-best passage in the worst seat, so the strongest material belongs at the two ends with the weaker candidates buried between them. The honest caveat is that the effect varies considerably by model and has narrowed as training on long inputs has improved. It is a consideration to test, not a law to design around.\nWhat accumulates has to be cut down A history that outgrows the window has to be shortened, and the standard response is compaction: summarize the older turns, replace them with the summary, carry on. It works, in the sense that the conversation continues and the recent material is intact. The interesting part is the cost.\nSummarization is lossy, which is unsurprising, and the loss is not uniform, which is the part that bites. What a summarizer discards is whatever looked unimportant at the moment of summarizing — and in a long task the detail that turns out to matter is frequently the one that looked incidental early, the offhand constraint in the second message, the error message that was worked around rather than fixed, the field name that appeared once. A summary that reads as a faithful account of the conversation can be missing precisely the thing needed at step forty. Every compaction is a bet that the model\u0026rsquo;s present judgement of relevance will hold for the rest of the task, and the bet is settled much later, silently, in the form of work redone.\nTool results are the usual culprit in an agent loop, more than the conversation itself. A single call can return ten thousand tokens of which four matter, and unlike a prompt the model wrote none of it and had no way to anticipate its size before asking. Three responses are available and all are crude: truncate, which risks cutting the relevant line; filter or project, returning only the fields the caller plausibly needs; or return a handle — write the payload somewhere and pass back an identifier and a summary, letting the model fetch more if it decides it needs to. This is where most of the practical work in these systems actually goes, and it is unglamorous enough that it tends to be described as plumbing rather than design.\nFigure 2. Three crude answers to a ten-thousand-token tool result.\nA pinch of math: the budget and the bill Write the window as $W$ and the components as $c_1 \\ldots c_n$ — instruction, tools, retrieved documents, history, request. The constraint is an inequality, not an objective:\n$$\\sum_{i} c_i \\le W$$with one term effectively fixed, since the current request is not optional. Everything called context engineering is a choice about how the remaining slack is divided, and no allocation improves one term without taking from another.\nThe second piece of arithmetic concerns the bill. Let $P$ be the invariant prefix, re-sent on every turn, and suppose each turn adds roughly $R$ tokens of request and response. At turn $t$ the model reads about $P + tR$ tokens, so over $T$ turns the total read is\n$$\\sum_{t=1}^{T} (P + tR) \\;=\\; TP + R\\,\\frac{T(T+1)}{2}$$which is quadratic in the number of turns. A conversation at constant response length gets steadily more expensive per turn, not by any inefficiency but because the history is re-read each time it grows. Caching the prefix removes the $TP$ term and leaves the quadratic. That quadratic is the thing worth carrying away, and it is close to the only firm result on offer here.\nFigure 3. The bill is quadratic in turns.\nInstructions and data arrive on one channel The model receives one undifferentiated stream of tokens. There is no privileged register for instructions and no flag marking a passage as reference material, so text that arrived as a retrieved document, a fetched web page or a tool result is available to be read as an instruction, and sometimes is. Prompt injection follows from this directly. It is not a defect of a particular model but a consequence of the architecture, and the defences — marking untrusted regions, restricting which tools may be called once untrusted content has entered, requiring confirmation before anything irreversible — are mitigations that raise the cost of an attack rather than fixes that remove it. No known method makes a model reliably ignore instructions embedded in text it has been asked to read.\nThe same honesty is owed to the practice as a whole, which has almost no theory in it. What is known is largely empirical, tied to particular models, and liable to go stale when the model changes; a prompt layout carefully tuned on one system frequently transfers poorly to the next, and the reasons are rarely legible. The field also disagrees with itself more than the confident tone of most advice suggests, because measuring whether a context change helped requires the same expensive evaluation machinery as any other claim about model behaviour — a task suite, repeated runs, cost accounted alongside quality — and in its absence a change that felt better on four examples becomes received wisdom. A good deal of what circulates as context engineering is at that level of evidence.\nWhat to carry forward Context engineering is allocation under three simultaneous facts: the window is a fixed ceiling, every component draws on that same ceiling, and attention within it is uneven, so presence in the context does not guarantee use. Those three make it an engineering problem with real trade-offs, and they are why a larger window relaxes the arithmetic without changing the structure. A working context holds a system instruction, tool definitions, retrieved documents, a history and the current request, and the cost is paid not once but on every turn, since the model has no memory between calls and the whole context is read again — the KV cache is what makes the re-reading affordable, and it rewards a stable prefix while punishing an edit near the beginning, which invalidates everything after it. That is the strongest argument for putting invariant material first, a systems argument rather than a claim about attention ; the separate observation that models attend least reliably to the middle of a long input argues for placing the strongest passages at both ends, though the effect is model-specific and has narrowed. When history outgrows the window it is compacted, and compaction is a bet that what looks unimportant now will stay unimportant, which in long tasks it often does not. Tool output is the larger practical problem, since one call can return ten thousand tokens of which four matter and the model neither wrote it nor knew its size, which is why truncation, projection and returning handles rather than payloads absorb most of the real effort. Underneath all of it sits the fact that instructions and data share one channel, so injection is structural rather than a bug, the defences are mitigations, and almost everything else here is empirical, model-specific, and held more firmly than it is known.\nThe figures in this article follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/articles/context-engineering/","summary":"\u003cp\u003eA team builds an \u003ca href=\"/articles/agents/\"\u003eagent\u003c/a\u003e\n that works beautifully for six steps and then begins repeating itself, and the diagnosis they reach for is that the model is not clever enough. They try a larger one. The repetition continues, because the problem was never the reasoning: by step twelve the original instruction sits forty thousand tokens back, sharing a window with three stale directory listings and the full text of a file the agent read once for a reason it can no longer recover. The name that has settled on the business of managing this is \u003cem\u003econtext engineering\u003c/em\u003e, which sounds like prompt writing with a grander vocabulary and is not quite that. What distinguishes it is that the decisions have measurable costs and trade against one another, which is what makes something an engineering problem rather than a set of habits. This piece is about the budget every component competes for, the ordering that the cache and the attention pattern argue about separately, and the losses that compaction and tool output quietly impose.\u003c/p\u003e","title":"Context Engineering — Deciding What the Model Gets to See"},{"content":"Accounts of what makes a language model fast tend to stop at the model. Cheaper attention kernels, fewer bits per weight, a small model guessing ahead — all of it aimed at making one forward pass cost less. Then they note, almost in passing, that serving is affordable because requests are batched, as though a batch were something you assemble and hand over. It is not. Requests arrive at times nobody chose and stop after wildly different numbers of tokens; one wants a yes or a no, the next a thousand-word summary. Nearly every difficulty in production serving descends from that variance, and the two mechanisms that deal with it are a scheduler and a memory allocator rather than anything inside the network. This piece is about how a batch is kept full when its members keep leaving, and about the allocator that decides how many members it may have.\nStatic batching wastes the machine twice Begin with the arrangement any reasonable engineer builds first. Requests are held until sixteen accumulate or a timer expires, run together from their prompts to their final tokens, and handed back. Simple, easy to reason about, and it discards most of the machine in two ways.\nThe padding is computed on. A batch is a rectangle, and prompts are not. Nine tokens, four hundred, eleven — the short ones are extended with filler to match the longest, and that filler travels through every layer, occupying arithmetic units and bandwidth, before being masked out. Real work performed on nothing. The waste is bounded, at least, since prompt lengths usually differ by a factor of a few.\nThe straggler sets the pace. Generation lengths are not so well behaved. One request stops after twenty tokens, another runs to two thousand, and the batch is not finished until its longest member is. Fifteen slots then hold finished sequences that still occupy memory and are still carried through every pass, producing nothing. Throughput is governed by the maximum rather than the mean, and when lengths span an order of magnitude the maximum of sixteen draws sits far above it. You pay for the longest answer sixteen times over and deliver the typical one.\nThe fix is to schedule tokens rather than requests The remedy is a change of granularity and nothing else: treat a single forward pass as the unit of scheduling rather than a whole request, letting the composition of the batch change from pass to pass. This is continuous batching, also called iteration-level scheduling.\nAfter every pass, any sequence that has emitted a stop token is retired, its slot released, and a waiting request admitted on the very next iteration. A table cleared and reseated between courses, rather than the whole sitting turning over at once. Nothing about the model changes, and the batch never drains.\nReported gains run to several times the throughput of static batching, with a qualification: the multiple is a property of the traffic, not the method. If every request produced the same number of tokens, static batching would already be near optimal. Under a real mixture of one-line replies, long summaries, and agent loops resending a growing transcript, the variance is large and so is the recovery.\nRefilling the batch creates a problem of its own. Admitting a request is not the same operation as continuing one. A new arrival\u0026rsquo;s prompt must be processed — many thousands of positions at once, one large parallel operation limited by how fast the chip can compute. Everything already running needs one token each: a thin serial step limited by how fast weights stream out of memory. Prefill and decode. Opposite shapes, one machine.\nMixing them into one batch is the easy implementation, and it has a visible cost. A prefill over a hundred thousand tokens — a long document, a context filled to its limit — occupies the machine for as long as many decode steps, and every running request stalls for the duration. Users see a stream that was flowing and then stopped. Separate pools for the two phases avoid the stall but leave whichever is under-loaded idle, and the cache must cross an interconnect when a request graduates. So the tension is managed rather than solved: chunked prefill slices a long prompt into pieces of a few hundred tokens, one per decode batch, spreading the delay thinly at the cost of a slower first token for the sliced request; disaggregation splits the pools and pays with a transfer and a provisioning guess. Neither makes the difference in shape go away.\nFigure 1. Static and continuous batching, slot by slot.\nNobody knows how long an answer will be The scheduler now needs an answer to a question it cannot get: how many sequences fit at once. What decides that is the key-value cache , since each layer stores a key and a value for every token seen, growing by one entry per token generated. The final size is unknown until generation stops, and generation stops when the model decides it does.\nThe obvious allocator therefore reserves the worst case. Each slot gets one contiguous region large enough for the longest response permitted, nothing ever moves, and the attention kernel reads a clean contiguous span, which is what such kernels were written for. The arrangement is also close to a disaster. A request that stops after fifty tokens has spent its whole life holding room for four thousand.\nQuoted plainly, it was common for well under half of reserved cache memory to be holding real data, and under a quarter was not unusual. This is internal fragmentation — waste inside each reservation rather than gaps between them — and it was the binding constraint on concurrency. The cache decides how many fit, how many fit decides throughput, and most of the memory set aside was empty. What is being squeezed here is the serving batch, a scheduling artifact rather than the statistical training batch size .\nThe answer was borrowed from operating systems The move that fixed it is not a new idea, and that is the interesting part. An operating system has faced the identical problem for decades: processes want contiguous address ranges, memory arrives in awkward gaps, and nobody knows how much any will want. The settled answer is to stop handing out contiguous memory — fixed-size pages, one at a time, and a table per process translating addresses to whichever pages hold them.\nPaged attention is that, applied to the cache. Cache memory becomes a pool of fixed-size blocks holding some tens of tokens each, sixteen being typical, each sequence carrying a table from logical positions to the physical blocks storing them. A block is allocated when the previous one fills and not sooner, anywhere in the pool. Waste per sequence collapses to the unused part of one partly-filled final block. The system that made both standard, from 2023, shipped scheduler and allocator together, because each is far less useful without the other.\nThe indirection turned out to be worth more than the space it saved. Because logical positions reach physical memory through a table, two tables can point at one block. Requests beginning with the same text — a shared system prompt, a few-shot preamble, the same document asked about repeatedly — have byte-identical cache contents there, so the prefix is stored and computed once, copying deferred until one diverges. The same holds for parallel samples from one prompt, common in decoding : the shared trunk exists once, not once per branch. This is the mechanism underneath the prompt-caching discounts providers advertise, and the discount is honest: the work genuinely was not repeated.\nThe costs land in three places. Every attention lookup now passes through a translation, so the kernel cannot stream a contiguous span and must gather from blocks scattered around the pool — which meant the fused kernels behind FlashAttention had to be rewritten to walk a block table while tiling. The block size is a trade with no comfortable answer: small blocks waste less inside but multiply table entries and overhead, large blocks the reverse. And the scheduler becomes intricate, since admitting sequences until memory is full guarantees it will sometimes run out mid-generation, so a running request must be preempted, its blocks reclaimed, its cache swapped to host memory or recomputed later. Eviction policies have bad days.\nNone of it touches the model. The arithmetic per token is unchanged, and so is the memory traffic that quantization attacks and the idle arithmetic that speculative decoding converts into extra tokens. Every generated token still costs a full read of every weight. A single request on an empty machine runs no faster under these schemes, slightly slower given the indirection and scheduling overhead. Throughput techniques in the strict sense: efficiency under load, and nothing whatever for the user waiting alone.\nFigure 2. One block, two tables: the shared prefix.\nA pinch of math: reserved memory against memory in use Suppose the machine holds $B$ slots, each reserving room for the maximum permitted length $L_{\\max}$, while the sequences in them reach actual lengths $L_1 \\ldots L_B$. Utilization of the reserved cache is then\n$$U = \\frac{\\sum_{i=1}^{B} L_i}{B \\, L_{\\max}} = \\frac{\\bar{L}}{L_{\\max}}$$the ratio of mean realized length to maximum allowed, depending on nothing else about the distribution. Put plausible numbers in. A cap of four thousand tokens against an average completion of four hundred gives one tenth: nine tenths of the cache reserved, holding nothing. Raising the cap to serve the occasional long answer lowers utilization for everybody, which is the perverse part.\nUnder paging the waste per sequence is the unused remainder of its final block, which for block size $b$ lies between zero and $b-1$ tokens and averages roughly half of $b$, so across $B$ sequences\n$$W \\le B\\,(b-1), \\qquad \\mathbb{E}[W] \\approx \\frac{B \\, b}{2}$$and the term that matters is the one that is absent. There is no $L_{\\max}$ in either expression. With blocks of sixteen tokens and sixty-four sequences the expected waste is a few hundred tokens, against the hundreds of thousands thrown away by reservation. That independence is the whole result: a generous length limit costs nothing until somebody uses it.\nFigure 3. Utilization is the mean length over the cap.\nWhat to carry forward Everything here follows from one property of the workload: requests arrive at times nobody chose and stop after token counts an order of magnitude apart. Batch them the obvious way — collect a group, run it to completion, hand it back — and the machine is wasted twice: lightly on padding for shorter prompts, heavily on slots that finished early and sit inert until the longest is done, so throughput is set by the straggler and not the average. Continuous batching removes the second waste by scheduling at the granularity of a forward pass rather than a request, admitting waiting requests between passes so the batch never drains; the gains of several times are real, but they measure the traffic\u0026rsquo;s variance as much as the technique. Refilling continuously creates its own difficulty: admitting a request means processing a whole prompt at once while continuing one means producing a single token, operations of opposite shape, so mixing them stalls every running stream when a long prompt arrives while separating them leaves one pool idle — a tension managed by chunking prompts or splitting machines rather than resolved. How many fit is set by the key-value cache, which grows one entry per token to a length nobody knows in advance, so reserving the maximum per slot left well under half, often under a quarter, holding nothing. Paged attention borrows the operating system\u0026rsquo;s answer — fixed-size blocks, a per-sequence table, allocation on demand — and since the indirection lets two tables point at one block, a shared prefix is stored and computed once. What it buys is utilization under load. The lone user on a quiet machine gains nothing, and the scheduler that makes the rest work must now be able to evict a sequence in the middle of its answer.\nThe figures in this article follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/articles/continuous-batching/","summary":"\u003cp\u003eAccounts of what makes a language model fast tend to stop at the model. Cheaper attention kernels, fewer bits per weight, a small model guessing ahead — all of it aimed at making one forward pass cost less. Then they note, almost in passing, that serving is affordable because requests are batched, as though a batch were something you assemble and hand over. It is not. Requests arrive at times nobody chose and stop after wildly different numbers of tokens; one wants a yes or a no, the next a thousand-word summary. Nearly every difficulty in production serving descends from that variance, and the two mechanisms that deal with it are a scheduler and a memory allocator rather than anything inside the network. This piece is about how a batch is kept full when its members keep leaving, and about the allocator that decides how many members it may have.\u003c/p\u003e","title":"Continuous Batching and Paged Attention — Keeping the Machine Fed"},{"content":"Every number in a model is where it is because of one quantity. The loss is what training minimizes, what scaling laws predict, what a training run is monitored by, and what all the machinery of optimization exists to descend. For a language model that quantity is cross-entropy, and it is usually introduced as a formula to be accepted rather than a thing to be understood. That is a shame, because it has an exact and rather beautiful interpretation — it is the number of bits you would waste encoding real text using the model\u0026rsquo;s beliefs about text — and once that reading is in hand, several otherwise mysterious facts about training curves and model evaluation become obvious. This piece is about what the loss measures, why it and not something simpler, and what its familiar cousin perplexity adds.\nThe problem with counting mistakes The obvious way to score a predictor is by accuracy: how often was the top choice correct? For a language model this is close to useless, and for two reasons.\nThe first is that it is nearly always wrong, and correctly so. Given the cat sat on the, a good model will say mat, but floor, sofa, step, and roof are all perfectly reasonable, and the text may well continue with any of them. A model that assigns mat forty percent and the alternatives sensible shares is behaving well. Accuracy scores it zero whenever the text happened to say floor. Worse, accuracy cannot distinguish that model from one that assigned mat a hundred percent and everything else nothing — a model that is far more confidently wrong, and far worse.\nThe second reason is fatal for training rather than merely for evaluation. Accuracy is a step function: it does not change at all as a probability moves from 0.1 to 0.4, then jumps the instant the ranking flips. A quantity that is flat almost everywhere has a gradient of zero almost everywhere, and gradient descent has nothing to descend. You cannot train on it.\nWhat is needed is a score that grades the whole distribution, rewards putting mass on what actually happened, punishes confident errors more than tentative ones, and varies smoothly so that any small improvement registers. Cross-entropy is that score, and it turns out to be essentially the only natural one.\nFigure 1. Accuracy is flat; surprise is not.\nSurprise, measured in bits Start from information rather than from statistics. Suppose you believe an event has probability $p$. When it occurs, how surprised should you be?\nWhatever the answer, it should have two properties. Something you were certain of should carry no surprise, and something you thought impossible should carry infinite surprise. And the surprise of two independent events together should be the sum of their surprises, since surprise ought to add the way information does — while their probabilities multiply. A function that turns multiplication into addition is a logarithm, and the unique measure satisfying both requirements is:\n$$\\text{surprise}(p) = -\\log p$$Take the logarithm base two and the unit is the bit, with a pleasing concrete meaning: it is the length, in binary digits, of the ideal codeword for an event of probability $p$. Something with probability one-eighth costs three bits to name. Something near-certain costs almost nothing. Something you assigned one chance in a million costs twenty bits — you must transmit a long message, because you had not prepared for this.\nThe loss of a language model is exactly this, averaged over every token in the data:\n$$L = -\\frac{1}{N} \\sum_{t=1}^{N} \\log p_\\theta(x_t \\mid x_{","permalink":"https://www.fanfani.net/articles/cross-entropy/","summary":"\u003cp\u003eEvery number in a model is where it is because of one quantity. The loss is what training minimizes, what scaling laws predict, what a training run is monitored by, and what all the machinery of \u003ca href=\"/articles/adamw/\"\u003eoptimization\u003c/a\u003e\n exists to descend. For a language model that quantity is cross-entropy, and it is usually introduced as a formula to be accepted rather than a thing to be understood. That is a shame, because it has an exact and rather beautiful interpretation — it is the number of bits you would waste encoding real text using the model\u0026rsquo;s beliefs about text — and once that reading is in hand, several otherwise mysterious facts about training curves and model evaluation become obvious. This piece is about what the loss measures, why it and not something simpler, and what its familiar cousin perplexity adds.\u003c/p\u003e","title":"Cross-Entropy — Measuring Wrongness in Bits"},{"content":"Almost all the attention paid to language models concerns the machine: how it is shaped, how it is optimized, how it is served. What goes into it is the part with the least published detail and quite possibly the largest effect. Two organizations can train the same architecture at the same scale with the same optimizer and produce models of visibly different quality, and the difference is the corpus. The decisions involved are not glamorous — what fraction should be code, how many times may a document be repeated, what to do with the internet\u0026rsquo;s vast supply of near-duplicate junk, what to feed the model in its final week — and they are made with far less theory than any other part of the pipeline. This piece is about what those decisions are and the surprisingly small amount that is actually known about them.\nFiltering: most of the internet is not worth reading The raw material is a web crawl, and a web crawl is mostly not prose. It is navigation menus, cookie banners, boilerplate footers, machine-generated listings, keyword spam, and text so degraded that no human would read it voluntarily. A common crawl snapshot reduces, after processing, to somewhere between one and ten percent of its original volume. The pipeline that performs the reduction is a sequence of increasingly expensive filters.\nThe cheap ones come first. Strip the markup. Detect the language and keep what you want. Apply heuristics that catch the obvious: documents too short to contain an idea, documents whose lines mostly lack terminal punctuation, documents with an implausible ratio of symbols to words, documents consisting largely of a repeated line. These rules are inelegant and remove an enormous amount of material for very little compute.\nThen deduplication, which matters more than its dullness suggests. The web repeats itself relentlessly — syndicated articles, mirrored documentation, quoted passages, templated pages differing in one field. Exact duplicates are easy. Near-duplicates require approximate matching, typically by hashing overlapping word sequences in a way that makes similar documents collide, then discarding all but one member of each cluster. Deduplication improves models substantially at fixed compute, and it does so through two mechanisms worth separating: it stops the model spending capacity memorizing text it has seen forty times, and it prevents contamination of held-out evaluations by material that also appears in training.\nLast comes quality classification, which is where judgement enters. The usual approach trains a small classifier to distinguish crawl text from some reference corpus assumed to be good — reference-quality encyclopedic articles, well-regarded link aggregators, curated book collections — and keeps the documents it scores highly. It works, and it is worth being honest about what it does. A classifier trained to recognize text resembling a particular reference is a machine for encoding whose writing counts as good, and the resulting corpus inherits every bias in that choice. More recently the classifier is often a language model asked to rate educational value directly, which is more flexible and no less a judgement call.\nFigure 1. A crawl reduced to a few percent.\nMixing: the proportions are the model Having assembled sources — filtered web text, code, books, scientific papers, reference works, mathematics, multilingual material, and now large volumes of synthetic text — you must decide how much of each the model reads. These proportions are among the most consequential numbers in the whole enterprise and among the least published.\nTwo findings are robust enough to state. The first is that code helps things that are not code. Models trained with a substantial fraction of source code perform better on reasoning tasks with no programming content. The explanation usually offered is that code is unusually well-structured text with long-range dependencies, explicit compositional structure, and a strong correspondence between form and consequence — a good curriculum for whatever underlies multi-step inference. The explanation is plausible and not proven; the effect is measured repeatedly.\nThe second is that quantity and quality trade against each other at a rate that depends on how much compute you have. Aggressive filtering leaves a smaller, better corpus. If the budget is small relative to the data available, filter hard, since you will not exhaust even the strict subset. If the budget is large enough that you would have to repeat the strict subset several times, the calculation changes and a looser filter that supplies fresh tokens may win. This interacts directly with the scaling arithmetic, and it is why the right filtering threshold is not a property of the filter but of the run.\nAttempts to set the proportions systematically exist. One approach trains a family of small models on different mixtures and fits a predictive relationship between mixture and loss, then optimizes it. Another treats the mixture as adjustable during training, reweighting toward sources where loss is still falling fastest. Both are real improvements over guessing and neither is decisive, and the proportions in a frontier model remain substantially a matter of accumulated institutional judgement.\nRepetition: how many times may a token be read If the corpus is smaller than the training budget, documents must be repeated. How much repetition is tolerable turns out to have a reasonably clean answer.\nUp to about four passes over the same data, the loss achieved is close to what an equivalent volume of fresh data would have given — a penalty of well under one percent. The repeated tokens are nearly as valuable as new ones. Returns then decay, with meaningful gains persisting to roughly sixteen passes and collapsing sharply after; by forty passes additional epochs contribute essentially nothing, and runs pushed that far can begin to degrade outright.\nThis has a direct consequence for planning, and it also frames the much-discussed question of running out of text. The supply of high-quality human-written material is finite and models are consuming it at a rate that has been extrapolated to exhaustion within a few years. The four-epoch result means the wall is further off than a naive count suggests, since the effective supply is several times the nominal one. But it is a multiplier, not a reprieve.\nFigure 2. Four passes are nearly free, forty are worthless.\nSynthetic data, and the thing to be careful about The response has been to generate text rather than find it, and it is now a large fraction of what goes into a model.\nThe useful cases are specific. Rewriting existing documents into cleaner or more instructive form. Generating problems and solutions in domains where correctness can be checked, which connects directly to verifiable rewards — a solution that passes its tests is a solution worth training on, and the checker means the data can be trusted without a human reading it. Producing coverage for situations that are rare in natural text. Distilling a stronger model\u0026rsquo;s outputs, which is distillation by another route.\nThe hazard is generating text with a model and training the next model on it without any external check. Each generation is sampled from the last, and sampling from a distribution and refitting to the samples loses the tails — rare constructions are drawn rarely, so they are under-represented in the sample, so the next model assigns them even less probability, so they are drawn less often still. Repeat this and the distribution narrows toward its own mode. The effect has been demonstrated cleanly in controlled settings, and the conditions under which it appears are the important part: it requires replacing real data with synthetic and closing the loop. Accumulating synthetic data alongside real data, or filtering it through any external criterion, largely prevents it. Which is why the practice that works pairs generation with verification, and the practice that fails is generation alone.\nOrdering: the model\u0026rsquo;s last week matters most The final decision is not what to include but when, and it is the one that has changed most recently.\nData order matters because a model trained by gradient descent is influenced most by what it saw last, especially as the learning rate decays toward the end of a run and each remaining step commits the weights more permanently. This has been turned into a deliberate technique. Rather than sampling uniformly from a fixed mixture throughout, training now commonly divides into a long bulk phase on the broad, cheap, heavily-filtered corpus, followed by a shorter annealing or midtraining phase in which the mixture shifts sharply toward the highest-quality material — textbooks, curated mathematics, verified code, instruction-like text — while the learning rate decays to near zero.\nThe gains from this final phase are large relative to its cost, which is a few percent of the total. It also fits neatly with the warmup-stable-decay learning rate schedule, whose long constant-rate middle allows training to be extended or stopped at will, with the expensive decay-and-anneal performed once at whatever endpoint is chosen. And it has quietly moved the boundary between pretraining and post-training: capabilities that were previously introduced by fine-tuning are now increasingly folded into the end of pretraining, where they cost less and stick better.\nFigure 3. The bulk phase, then the last week.\nWhat to carry forward A model is largely made of what it read, and the decisions governing that are the least documented and possibly the most consequential in the pipeline. Raw web text survives filtering at a rate of a few percent, through cheap heuristics first, then deduplication — which improves models both by preventing wasted capacity on repeated text and by keeping evaluations uncontaminated — and finally quality classification against a reference corpus, a step that unavoidably encodes somebody\u0026rsquo;s judgement about what good writing is. Of the mixing proportions, two things are reliably known: a substantial share of code improves reasoning on tasks unrelated to code, and the right filtering strictness depends on the compute budget, since aggressive filtering is correct only when you will not be forced to repeat the survivors. Repetition itself is nearly free to about four passes, still worthwhile to around sixteen, and worthless by forty — which multiplies the effective supply of human text without removing the limit. Synthetic data addresses the shortage and works when paired with an external check, such as a test suite or a stronger teacher; the failure mode requiring care is a closed loop in which real data is replaced by a model\u0026rsquo;s own samples, which narrows the distribution generation by generation as the tails go unsampled. And ordering matters most at the end: a final annealing phase on the highest-quality material, as the learning rate decays to zero, buys a disproportionate share of the model\u0026rsquo;s quality for a few percent of its cost, and has been steadily absorbing work that used to belong to fine-tuning.\nThe figures in this article follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/articles/data-mixtures/","summary":"\u003cp\u003eAlmost all the attention paid to language models concerns the machine: how it is shaped, how it is optimized, how it is served. What goes into it is the part with the least published detail and quite possibly the largest effect. Two organizations can train the same architecture at the same scale with the same optimizer and produce models of visibly different quality, and the difference is the corpus. The decisions involved are not glamorous — what fraction should be code, how many times may a document be repeated, what to do with the internet\u0026rsquo;s vast supply of near-duplicate junk, what to feed the model in its final week — and they are made with far less theory than any other part of the pipeline. This piece is about what those decisions are and the surprisingly small amount that is actually known about them.\u003c/p\u003e","title":"Data Mixtures — Deciding What a Model Reads, and When"},{"content":"A model never produces a word. What comes out of the tower is a set of scores, one per vocabulary entry, and after the softmax a set of odds — a forecast, not a decision. Something outside the model has to read that forecast, commit to a single token, and then do it again, and again, until the text is finished. That something is the decoding strategy, and it is governed by a handful of dials that appear in almost every interface and API you will meet. This piece is about what each of them does, why several of them exist to solve the same problem in different ways, and why not one of them makes a model smarter.\nThe one thing none of these dials do It is worth being blunt about this at the outset, because a great deal of confusion follows from getting it wrong. By the time any of these settings takes effect, the model has finished. The forward pass is complete, the distribution over the next token is fixed, and nothing downstream will cause the model to reconsider. Temperature does not make it more imaginative. Truncation does not make it more careful. Every one of these dials is a policy for reading a forecast that has already been issued.\nA forecast of seventy percent rain does not tell you whether to carry an umbrella. That depends on how much you mind getting wet against how much you mind carrying an umbrella, and reasonable people with the same forecast make different choices. Decoding parameters are that second decision, and they should be judged the way you would judge any decision rule: not by whether they improve the forecast, which they cannot, but by whether they suit what you are about to do with it.\nTwo consequences follow, and both matter more than they first appear.\nThe first is that any deviation from plain sampling is a deliberate distortion. A well-trained model\u0026rsquo;s distribution is roughly calibrated — it was trained, after all, to match the statistics of real text — so sampling from it exactly is in one precise sense the correct thing to do. We nonetheless almost never do it, because a model that is right about the shape of the distribution can still be wrong in the tail, and because most of the time we want good text rather than representative text. That is a defensible trade, but it is a trade. Nobody is getting something for nothing.\nThe second is that errors compound. Each token conditions everything after it, so an unfortunate choice does not merely produce one odd word — it leaves the model in a slightly odd state from which the next choice is drawn, and the next. This is made worse by the fact that a model is trained on human text and then asked to continue its own, a discrepancy that widens with every token it generates. Which is why decoding settings barely matter for a three-word extraction and matter enormously for two pages of prose.\nGreedy, and why it is worse than it sounds The simplest possible rule is to take the highest-scoring token every time. It is deterministic, it is reproducible, and it sounds like it ought to be optimal. It is neither optimal nor, for most purposes, good.\nThe shallow objection is the familiar one about greedy algorithms: the best first step need not lie on the best path. A token that is marginally the most likely here may lead into a region where every continuation is poor, while the runner-up opened onto something far better. Choosing locally and choosing globally are different problems.\nThe deeper objection is stranger, and it took the field some time to articulate. The most probable sequence is not a typical sequence. Real human text carries a characteristic amount of surprise — writers do not choose the single most predictable word at every opportunity, and text that does so does not read as excellent, it reads as inhuman. Push a distribution over sequences toward its mode and what you find at the peak is very often something nobody would ever write.\nIn practice this shows up as repetition. Greedy decoding on an open-ended prompt has a marked tendency to fall into a loop, repeating a phrase or a sentence indefinitely, and the reason is a feedback effect the model has honestly learned. Its training data is full of genuine repetition — lists, tables, structured records, code — so it has learned, correctly, that a pattern which has occurred twice is quite likely to occur a third time. Once a phrase has been emitted twice by accident, that machinery fires, the phrase becomes the most probable continuation, and greedy decoding is obliged to take it. Each repetition makes the next more likely. There is no exit.\nNone of which makes greedy decoding useless. Where there is a canonical right answer — extracting a field, classifying a document, producing code with a conventional form — reproducibility is worth a great deal and the failure mode never arises. It is open-ended generation where it collapses.\nFigure 1. The loop the model honestly learned.\nTemperature: flattening and sharpening the field The remedy is to sample rather than to maximize, and the first dial governs how the sampling is weighted. Temperature divides every logit by a constant before the softmax, which is the same operation described in the article on logits and the same one used, to a completely different end, to soften a teacher\u0026rsquo;s output in distillation .\nBelow one, the gaps between scores widen, the leader pulls further ahead, and the distribution sharpens toward the greedy limit. Above one, the gaps close, the field bunches, and at sufficiently high values the model is choosing more or less at random from its vocabulary. Zero is greedy exactly.\nIt is routinely described as a creativity dial and that description is misleading. Raising the temperature does not supply the model with ideas it did not have; the ranking of candidates is unchanged and no new option is invented. What changes is how far down the existing ranking you are willing to reach. Sometimes the token four places down was the interesting one, and the result reads as inspiration. More often it was ranked fourth for a good reason, and what follows is the model gracefully writing its way around a word it would not have chosen. Both outcomes are the same mechanism.\nTwo practical notes. Whether temperature is applied before or after truncation differs between implementations, and the two orders give different results — scaling first and then taking the top share of the mass is not the same as taking the top share and then scaling it. It is worth knowing which your stack does. And reasoning models frequently want a specific moderate temperature rather than the lowest available, because they were trained on rollouts sampled at that temperature; running them greedily can measurably degrade them, which surprises people who assume that less randomness is always safer.\nTruncation: top-k, top-p, min-p Temperature reweights the whole vocabulary but excludes nothing, and exclusion turns out to be the more important job.\nThe reason is arithmetic. A vocabulary has on the order of a hundred thousand entries, and the overwhelming majority of them are absurd in any given context. But absurd is not the same as impossible: assign each a probability of a millionth and the tail collectively holds a non-trivial share of the mass. A small per-token chance of drawing something ridiculous becomes, over a thousand tokens, a near-certainty of drawing it at least once — and by the argument above, one such token does not stay local. Truncation exists to convert \u0026ldquo;unlikely\u0026rdquo; into \u0026ldquo;impossible,\u0026rdquo; which is a categorically different guarantee.\nThree rules are in common use. Top-$k$ keeps a fixed number of the highest-scoring candidates. Top-$p$, also called nucleus sampling, keeps as many candidates as are needed to account for a given share of the probability, so the number kept adapts to the shape of the distribution. Min-$p$ keeps every candidate whose probability is at least some fraction of the leader\u0026rsquo;s. A fourth, locally typical sampling, is the odd one out: rather than working down from the top, it keeps tokens whose surprise is close to the distribution\u0026rsquo;s average surprise, on the theory that human text maintains a fairly steady information rate — which means it can discard the most likely token for being too predictable.\nA pinch of math: what the three rules actually hold constant All three do the same two things — keep a subset of the vocabulary, discard the rest, and renormalize what survives so it is once again a set of odds summing to one. They differ only in how the subset is chosen. Order the vocabulary by probability so that $p_1 \\geq p_2 \\geq \\ldots$, and the definitions sit comfortably side by side:\n$$S_{\\text{top-}k} = \\{1, \\ldots, k\\}$$$$S_{\\text{top-}p} = \\{1, \\ldots, m\\} \\ \\text{ where } m \\text{ is the smallest index with } \\sum_{i \\leq m} p_i \\geq p$$$$S_{\\text{min-}p} = \\{\\, i \\;:\\; p_i \\geq \\tau \\cdot p_1 \\,\\}$$Read them as three different quantities being held fixed. Top-$k$ fixes the count — exactly $k$ candidates, whatever shape the distribution happens to have. Top-$p$ fixes the mass — however many candidates it takes to reach a given share of the probability. Min-$p$ fixes a ratio to the leader — everything within some factor of the front-runner, and nothing else.\nThe difference is easiest to see in the case that breaks the first two. Suppose the model is confident: one token at $0.9$, with the remaining tenth of the probability smeared across ten thousand tail entries. Top-$k$ with $k = 40$ admits thirty-nine tail tokens that are collectively nonsense. Top-$p$ with $p = 0.95$ is worse than it looks, because it must keep adding candidates until the running total reaches $0.95$ — and since the leader supplies only $0.9$, it goes on to admit however many thousands of tail tokens are needed to scrape together the missing $0.05$. Min-$p$ with $\\tau = 0.1$ admits tokens at $0.09$ or above, which here means the leader alone.\nNow flip it. Suppose the model is genuinely torn between two hundred reasonable continuations, each near $0.005$. Top-$k$ discards a hundred and sixty of them for no reason at all. Top-$p$ behaves well. And min-$p$, its threshold now standing at a mere $0.0005$, opens up and lets nearly all of them through.\nThat is the whole argument for the newest of the three: it tightens automatically when the model is sure and loosens when it is not, because it is the only one that measures anything relative to how peaked the distribution already is. All of them are crude proxies for the quantity genuinely at issue, which is the entropy of the distribution — how much real uncertainty the model is expressing at this position — and none of them measure it directly.\nFigure 2. What each truncation rule holds fixed.\nPenalties, and why they are blunt A separate family of dials attacks repetition head-on by penalizing tokens that have already appeared. A repetition penalty scales down the logit of any token seen so far. A frequency penalty subtracts an amount proportional to how many times it has appeared. A presence penalty subtracts a flat amount the moment it appears at all.\nThey work, in the narrow sense that they reduce repetition. The trouble is that they operate on the identity of individual tokens rather than on the phenomenon anyone actually objects to, which is a degenerate loop. The word \u0026ldquo;the\u0026rdquo; will appear a hundred times in any decent paragraph and deserves no penalty whatsoever. A variable name in a program must be repeated exactly, and punishing it produces code that does not run. A list of similar items is repetitive on purpose. These penalties cannot tell any of that apart from a model stuck in a loop, so they tax the good with the bad, and turning them up far enough to stop a determined loop will visibly distort ordinary prose.\nThere is also a genuine implementation wart worth knowing about. The classic repetition penalty is applied multiplicatively, dividing a seen token\u0026rsquo;s logit by a factor greater than one. That reduces a positive logit, as intended — but a negative logit divided by a number greater than one moves toward zero, making the token more likely rather than less. The original formulation had to special-case the sign, and implementations that did not do so shipped with the penalty working backwards for exactly the tokens the model was least keen on.\nThe modern view is that these are patches for a symptom. A capable model with a well-chosen truncation rule loops far less than a weak one, and needs far less penalizing. They remain useful for small models and for particular formats, but reaching for them first is usually a sign that something upstream is wrong.\nBeam search, and why chat models abandoned it Every rule so far commits to one token and moves on. Beam search does not: it keeps several partial sequences alive at once, extends each by every plausible next token, scores the resulting continuations by their total probability, and prunes back to the best handful. It is an attempt to approximate the sequence the model considers most probable overall, rather than the one produced by choosing greedily at each step.\nIt needs one correction to work. A sequence\u0026rsquo;s probability is a product of terms below one, so it falls monotonically with length and the search would otherwise always prefer to stop early. Dividing the score by the length, raised to some power, is the standard fix, and that exponent becomes yet another dial.\nFor a certain class of tasks beam search is genuinely the right tool, and it remains standard in machine translation, speech transcription, and constrained summarization. What those share is that there is approximately one correct output, the model\u0026rsquo;s probability really does track quality, and the whole exercise is to find the best version of a fixed thing.\nFor open-ended generation it fails, and the way it fails is instructive. Beam search is not bad at its job. It is good at its job — it reliably finds high-probability sequences — and high-probability open-ended text is precisely the bland, hedging, repetitive material identified earlier as the mode of the distribution rather than a typical draw from it. The better the search, the more thoroughly it locates exactly what you did not want. That is a rare and clarifying kind of failure: the objective was wrong, and improving the optimizer only got you there faster. It is also expensive, multiplying both the computation and the cache by the number of beams.\nThe dials that are not about choice Several settings live alongside these without being decision rules at all, and lumping them together causes trouble.\nMaximum tokens and stop sequences are control flow. They decide when to halt, not what to say, and the first will cheerfully truncate a sentence in the middle. Stop sequences are matched against text rather than tokens, so a stop string that straddles a token boundary can be missed entirely.\nThe seed is more interesting than it looks, because fixing it does not actually guarantee identical output from a hosted model. Floating-point addition is not associative, so the order in which partial sums are combined changes the last bits of the result; that order depends on how work is grouped across the batch, and the batch depends on which other requests happened to arrive at the same moment. Two runs with the same seed can therefore diverge, usually harmlessly, occasionally at a position where two candidates were nearly tied. Reproducibility here is best-effort, not a guarantee.\nSpeculative draft length — how many tokens a small model proposes before the large one checks them — is a pure throughput dial. As described in the article on the KV cache , the verification step provably preserves the large model\u0026rsquo;s output distribution exactly, so this setting trades wasted draft work against speedup and has no effect whatsoever on quality.\nAnd then there is the thinking budget offered by reasoning models , which belongs in a category of its own. Every other dial here is a different way of reading one distribution. This one causes more distributions to be produced — more tokens of deliberation before the answer begins — and it is the only setting on the list that genuinely buys capability rather than trading one property for another. That it appears in the same settings panel as temperature is an accident of interface design, not a family resemblance.\nFigure 3. Sorting the settings panel by what it changes.\nWhat to carry forward A model\u0026rsquo;s output is a distribution, never a word, and every decoding parameter is a policy for collapsing that forecast into a choice — none of them changes what the model believes, and none can add information that was not in the distribution already. Plain sampling is in a strict sense the faithful thing to do, so every dial that departs from it is trading calibration for something else, and because each token conditions the next, those departures compound over a long generation. Greedy decoding fails on open-ended text not through bad luck but because the most probable sequence is not a typical one, and because the model\u0026rsquo;s honest belief that repeated patterns repeat again becomes a loop with no exit. Temperature rescales the gaps between scores and controls how far down the ranking you will reach; it invents nothing. Truncation is the more consequential intervention, since it converts unlikely into impossible, and the three common rules differ in what they hold fixed — top-$k$ a count, top-$p$ a share of the mass, min-$p$ a ratio to the leading candidate, with only the last automatically tightening when the model is confident. All are rough stand-ins for the entropy nobody measures directly. Penalties suppress repetition by taxing token identity rather than the pathology, so they punish a variable name and a stuck loop alike, and are best understood as patches for a weak model or a poor truncation setting. Beam search still wins where one right answer exists, and fails at open generation exactly because it succeeds at finding high-probability text. And of the remaining settings, most are control flow or throughput; the thinking budget is the sole exception, and the only dial in the panel that makes the model better rather than merely different.\nThe figures in this article follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/articles/decoding/","summary":"\u003cp\u003eA model never produces a word. What comes out of the tower is a set of scores, one per vocabulary entry, and after the softmax a set of odds — a forecast, not a decision. Something outside the model has to read that forecast, commit to a single token, and then do it again, and again, until the text is finished. That something is the \u003cem\u003edecoding\u003c/em\u003e strategy, and it is governed by a handful of dials that appear in almost every interface and API you will meet. This piece is about what each of them does, why several of them exist to solve the same problem in different ways, and why not one of them makes a model smarter.\u003c/p\u003e","title":"Decoding — Choosing a Word the Model Only Suggested"},{"content":"Nearly every language model in use writes the same way: one token, then the next, each conditioned on everything before it and on nothing after. That constraint is so fundamental it is easy to mistake for a fact about language rather than a choice about architecture. It is a choice, and it has costs — a model cannot revise a word it has already committed to, cannot plan the end of a sentence before writing its beginning, and must spend one full forward pass per token no matter how obvious the token is. Image generation abandoned sequential production years ago in favour of starting with noise and refining the whole canvas repeatedly. This piece is about what happens when the same idea is applied to text: why the translation is harder than it looks, what the working version actually does, and why the results are interesting without yet being decisive.\nWhy the image recipe does not transfer Diffusion over images works by adding a little Gaussian noise at a time until the picture is static, then training a network to reverse one step of that process. The whole construction rests on the data being continuous: you can add half a unit of noise to a pixel and get a slightly different pixel, and the path from a clean image to pure noise is a smooth one along which every intermediate point is a valid image.\nText is discrete. There is no token halfway between \u0026ldquo;cat\u0026rdquo; and \u0026ldquo;dog\u0026rdquo;, and adding Gaussian noise to a token index produces something that is not a token at all. The smooth corruption path that diffusion needs does not exist.\nThe obvious workaround is to run the diffusion in the continuous space of embeddings , where vectors can be noised, and round back to tokens at the end. This was tried extensively and works poorly. The embedding space is mostly empty — real tokens occupy a sparse scatter of points, and a noised vector lands in the vast space between them, where the rounding step is arbitrary and small differences in the vector produce completely different tokens. The geometry that makes diffusion work for images is absent.\nCorrupting by masking instead The construction that works replaces noise with erasure. Rather than degrading a token toward static, degrade it toward absent.\nThe forward process replaces each token independently with a special mask symbol, with a probability that rises from zero to one over the corruption schedule. At the start the sentence is intact; partway through, some fraction is masked; at the end everything is masked and no information remains. This is a valid corruption path, it is entirely discrete, and every intermediate state is a legitimate partially-observed sequence.\nThe reverse process is then: given a partially masked sequence, predict what belongs in the masked positions. The model is trained on exactly that — sample a masking rate, apply it, and ask the model to recover the original at every masked position, scored with ordinary cross-entropy .\nAnyone who has read about encoder models will recognize this immediately. It is masked language modelling, the objective that trained BERT, with one crucial change: BERT masked a fixed small fraction, around fifteen percent, while this masks a variable fraction drawn across the whole range from nothing to everything. That single difference converts a representation-learning objective into a generative one, because a model that can fill in a sequence which is one hundred percent masked is a model that can write from nothing.\nGeneration runs the process backwards. Start with everything masked. Run the model, which produces a distribution for every position at once. Unmask some of them — typically those the model is most confident about — and leave the rest masked. Run again, now conditioned on what was just filled in. Repeat until nothing is masked. The whole sequence is produced in a number of steps that is chosen rather than dictated by the length.\nFigure 1. Unmasking a sentence over four rounds.\nWhat this buys Four properties follow from generating in parallel, and they are genuinely different from what an autoregressive model offers.\nSteps are decoupled from length. A sequence of a thousand tokens can be produced in fifty passes rather than a thousand. Whether that is faster depends on the parallelism available, but the scaling is different: doubling the output length need not double the number of model invocations.\nRevision is possible. A token filled in early can be masked again in a later round and reconsidered in the light of what has since been written. Autoregressive generation has no mechanism for this at all — a committed token is permanent, and everything after it must be written around any mistake. This is the most structurally significant difference.\nConditioning is bidirectional and arbitrary. Filling a gap in the middle of a document, with fixed text on both sides, is the native operation here rather than an awkward special case. Infilling, editing, and constrained generation with fixed anchors all fall out of the training objective directly.\nPlanning is not sequential. Because the model sees the whole sequence at every step, it can settle the shape of a sentence before its contents, which is closer to how the constraint-satisfaction aspects of writing actually feel — and it is measurably better on tasks with strong global structure, such as puzzles whose constraints span the whole answer.\nA pinch of math: the cost of unmasking in parallel The central difficulty has a clean statement, and it explains why the step count cannot simply be lowered to one.\nAt each round the model produces a distribution for every masked position independently, conditioned on what is currently visible. If we unmask $k$ positions at once, we are sampling each from its own marginal:\n$$\\prod_{i \\in S} p(x_i \\mid x_{\\text{visible}})$$But the correct object is the joint distribution over those $k$ positions:\n$$p(x_{i_1}, \\ldots, x_{i_k} \\mid x_{\\text{visible}})$$and these are equal only if the positions are conditionally independent, which adjacent tokens in a sentence emphatically are not. Consider the blanks in \u0026ldquo;I\u0026rsquo;ll have a __ __\u0026rdquo; where the model\u0026rsquo;s marginals put weight on both \u0026ldquo;hot dog\u0026rdquo; and \u0026ldquo;cold beer\u0026rdquo;; sampling independently permits \u0026ldquo;hot beer\u0026rdquo;, which neither marginal disfavours and the joint would have ruled out.\nThis is exactly the cost of parallel generation, and it is why the number of steps matters. Unmask one position per round and you have recovered autoregression, with the joint respected exactly and no speed gain. Unmask everything in one round and you get incoherence. In between, every round conditions on what the previous round revealed, which repairs the dependencies incrementally — more rounds, better joint, slower generation. The quality-speed dial in a diffusion language model is precisely a dial on how much conditional dependence you are willing to ignore per step, and the strategy of unmasking the most confident positions first is a heuristic for choosing the ones where the marginal is closest to the joint.\nFigure 2. Marginals versus the joint when unmasking together.\nFigure 3. Steps as a dial on ignored dependence.\nWhere it stands Models of this kind now exist at meaningful scale — several billion parameters, trained on trillions of tokens — and they are close to autoregressive models of comparable size on standard benchmarks without matching the best of them. That is a substantial achievement given how much less total effort has gone into the approach, and it is not yet a reason to switch.\nTwo honest problems remain. The first is the KV cache , or rather its absence. Autoregressive generation caches keys and values because the past never changes; in a diffusion model every round can alter any position, so the cache is invalid and the whole sequence must be reprocessed each round. A fifty-step generation is fifty full passes over the entire sequence, which erodes much of the theoretical speed advantage. Approximate caching schemes that reuse work for positions unlikely to change are an active area and are, unavoidably, approximations.\nThe second is that the entire ecosystem assumes autoregression. Speculative decoding, cache management, the serving stacks, the post-training recipes — all of it is built for one token at a time, and a diffusion model inherits none of it.\nThe most likely near-term outcome is not replacement but combination: diffusion-style parallel drafting with autoregressive verification, or hybrid models that produce blocks in parallel and sequence the blocks. What the line of work has already established is worth keeping regardless of where it lands — that left-to-right generation is a design decision rather than a necessity, that its inability to revise is a real cost rather than an acceptable one, and that the tokens-equals-forward-passes equation can be broken.\nWhat to carry forward Diffusion language models abandon left-to-right generation for a process that produces the whole sequence at once and refines it over a chosen number of rounds. The image recipe does not transfer directly, because text is discrete and there is no token halfway between two tokens; running the diffusion in embedding space fails because real tokens are a sparse scatter and a noised vector lands in the empty space between them. The construction that works replaces noise with masking — corrupt a sequence by replacing tokens with a mask symbol at a rate swept across the full range from none to all, and train the model to recover them, which is masked language modelling with a variable rather than fixed rate, and that one change makes the objective generative. Generation starts fully masked and unmasks progressively, usually the most confident positions first. What this buys is real: steps decoupled from output length, the ability to revise a token already written, native infilling with fixed text on both sides, and non-sequential planning that helps on globally constrained tasks. The central cost is precise — unmasking several positions at once samples each from its own marginal while correctness requires the joint, and those differ whenever tokens are dependent, which is why fewer steps means more incoherence and why the step count is exactly a dial on how much conditional dependence is being ignored. Current models are close to autoregressive ones of similar size without surpassing them, they cannot use a KV cache because any position may change in any round, and they inherit none of the ecosystem built around one-token-at-a-time generation — which makes hybrid arrangements the likely near-term outcome, and leaves the durable contribution as the demonstration that sequential generation was a choice all along.\nThe figures in this article follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/articles/diffusion-lms/","summary":"\u003cp\u003eNearly every language model in use writes the same way: one token, then the next, each conditioned on everything before it and on nothing after. That constraint is so fundamental it is easy to mistake for a fact about language rather than a choice about architecture. It is a choice, and it has costs — a model cannot revise a word it has already committed to, cannot plan the end of a sentence before writing its beginning, and must spend one full forward pass per token no matter how obvious the token is. Image generation abandoned sequential production years ago in favour of starting with noise and refining the whole canvas repeatedly. This piece is about what happens when the same idea is applied to text: why the translation is harder than it looks, what the working version actually does, and why the results are interesting without yet being decisive.\u003c/p\u003e","title":"Diffusion Language Models — Writing a Sentence All at Once"},{"content":"The machinery that makes a large model good is well enough known: attention letting every word survey every other, a tall tower of blocks, and a long, expensive training run that settles billions of knobs into a capable configuration. Capability comes at a price. The best models are enormous, and every answer burns memory and time in proportion to their size — tolerable in a demonstration, punitive across billions of requests. So the wish is obvious: keep most of what a giant model knows while paying for a small one. Distillation is the technique that grants it, by transferring the large model\u0026rsquo;s judgement into a smaller one. This piece is about how the transfer works, why it works better than the size difference suggests, and what the student inherits along with the skill.\nThe problem with simply training a small model The obvious approach skips the big model entirely and trains a small one from scratch on the same data. Sometimes that is good enough. But a small model trained this way tends to plateau below what its size would allow — it has the raw capacity to be better and never gets there. The reason is subtle, and it is the key to everything that follows.\nOrdinary training presents a piece of text, asks for the next word, and scores the answer against the single word that actually came next. That correct word is treated as the whole truth and every other word as equally wrong. This throws away an enormous amount of information. Take the sentence \u0026ldquo;I poured milk into my\u0026hellip;\u0026rdquo;. The real next word might be \u0026ldquo;coffee\u0026rdquo;. Ordinary training says: \u0026ldquo;coffee\u0026rdquo; is right, and \u0026ldquo;tea\u0026rdquo;, \u0026ldquo;cereal\u0026rdquo;, \u0026ldquo;cup\u0026rdquo; and \u0026ldquo;airplane\u0026rdquo; are all simply wrong, with no distinction among them. But \u0026ldquo;tea\u0026rdquo; and \u0026ldquo;cereal\u0026rdquo; are nearly right, \u0026ldquo;cup\u0026rdquo; is plausible, and \u0026ldquo;airplane\u0026rdquo; is absurd. That texture — the relative plausibility of all the wrong answers — is real knowledge about the world. Standard training discards it.\nThe teacher\u0026rsquo;s real gift: the shape of its uncertainty Here is the insight that makes distillation work. A well-trained large model does not discard that texture. Asked to predict the next word it hands back a full set of odds across every possible word — a high number on \u0026ldquo;coffee\u0026rdquo;, smaller but respectable numbers on \u0026ldquo;tea\u0026rdquo; and \u0026ldquo;cereal\u0026rdquo;, a tiny number on \u0026ldquo;airplane\u0026rdquo;. These graded odds are the soft targets, and they encode more than the answer: they encode how the model has organized the whole landscape of possibilities around it. The relative sizes of the wrong answers are the organization made legible — that tea and coffee are cousins, that airplanes have no place in a cup.\nDistillation\u0026rsquo;s core move is to stop teaching the small model from the hard, one-word truth and teach it from the teacher\u0026rsquo;s soft odds instead. The lesson changes from \u0026ldquo;the answer is coffee\u0026rdquo; to a ranking over the whole vocabulary in which coffee leads, tea and cereal follow, and airplane is ruled out. The standard names make the roles plain: the big model is the teacher, the small model is the student, and the student imitates not the teacher\u0026rsquo;s final choice but the full distribution behind it.\nA pinch of math: turning scores into odds One piece of machinery makes this precise, and it is a familiar one. At the top of its tower, a model does not produce odds directly; it produces a raw score — a logit — for every possible word. Call these scores $z_1, z_2, \\ldots$, one per word in the vocabulary. The softmax turns them into odds that sum to one:\n$$p_i = \\frac{e^{z_i}}{\\sum_j e^{z_j}}$$Each score is exponentiated and divided by the total, so larger scores become larger probabilities and everything adds up to one — arithmetic you could do on the back of an envelope for a vocabulary of five. Nothing unusual — this is how every model turns its final scores into a guess.\nThe trouble is that a confident teacher produces a very spiky softmax. If \u0026ldquo;coffee\u0026rdquo; scores far above everything else, almost all the probability lands on \u0026ldquo;coffee\u0026rdquo; and the interesting texture — the small differences between \u0026ldquo;tea\u0026rdquo;, \u0026ldquo;cereal\u0026rdquo; and \u0026ldquo;airplane\u0026rdquo; — is squashed near zero, where the student cannot see it. The whole gift of distillation lives in those small numbers. Something has to bring them out.\nTemperature: softening the odds so the student can see them The fix is a single dial, the temperature, written $T$. Every score is divided by $T$ before the softmax:\n$$p_i = \\frac{e^{z_i / T}}{\\sum_j e^{z_j / T}}$$At $T = 1$ this is the ordinary softmax. Raise $T$ above one and the scores are pushed closer together, which softens the distribution — the spike over \u0026ldquo;coffee\u0026rdquo; comes down and the small numbers over \u0026ldquo;tea\u0026rdquo; and \u0026ldquo;cereal\u0026rdquo; rise into view. Distillation runs teacher and student at the same raised temperature, so the student trains against a spread distribution rather than a hard spike. The name is borrowed from physics, where high temperature spreads a system out and low temperature freezes it into place, and the intuition carries over unchanged.\nFigure 1. What temperature does to the teacher\u0026rsquo;s odds.\nMeasuring how well the student is imitating The student needs a score for how closely its own soft odds match the teacher\u0026rsquo;s. The yardstick is the Kullback–Leibler divergence, written $\\mathrm{KL}$, which measures how far one distribution sits from another. With the teacher\u0026rsquo;s odds $p$ and the student\u0026rsquo;s $q$:\n$$\\mathrm{KL}(p \\parallel q) = \\sum_i p_i \\log \\frac{p_i}{q_i}$$The divergence is zero when the two distributions are identical and grows as they drift apart, and it weights each term by $p_i$ — the teacher\u0026rsquo;s own probability. So it punishes the student hardest for disagreeing exactly where the teacher was most confident, and barely at all where the teacher had nothing to say. That direction is a choice with consequences, and the article on KL divergence sets them out: forward KL is mass-covering, so the student is pushed to reproduce the teacher\u0026rsquo;s whole distribution, wrong answers included. This term, computed on the temperature-softened odds, is the soft loss.\nTwo lessons at once: the combined loss The hard truth is not usually thrown away, because the word that actually came next is still reliable information. So the student trains on two lessons blended. The first is the ordinary one: match the single correct word, as in normal training — the hard loss. The second is the distillation lesson: match the teacher\u0026rsquo;s softened odds — the soft loss just built. A mixing weight $\\alpha$ between zero and one combines them:\n$$L = \\alpha \\, L_{\\text{hard}} + (1 - \\alpha) \\, L_{\\text{soft}}$$The dial $\\alpha$ decides how far the student trusts the cold, correct answer against the teacher\u0026rsquo;s richer but second-hand judgement. Lean toward the hard loss and the student behaves like an ordinary trained model; lean toward the soft loss and it tracks the teacher. Dividing scores by $T$ shrinks the gradients that drive learning by roughly a factor of $T^2$, so the soft loss is multiplied back by $T^2$ to keep the two pulling with comparable strength. It is a bookkeeping correction, not an idea. The recipe is then: run teacher and student warm, ask the student to imitate the softened odds while also getting the plain answer right, and descend on the blended loss with the same machinery any other model is trained with.\nFigure 2. The two lessons that make the blended loss.\nWhy it works: learning the reasons, not just the answers Step back and the reason distillation succeeds is visible. Ordinary training supplies a correct outcome and nothing else. Distillation supplies the teacher\u0026rsquo;s whole ranking — how sure it was, what near-misses it entertained, what it dismissed. The teacher\u0026rsquo;s uncertainty is its most instructive output: a compressed map of how the possibilities relate, handed over on every example at no extra cost once the teacher has run.\nThis is why a distilled small model routinely beats a same-sized model trained from scratch. Both have identical raw capacity. The distilled one was taught with richer lessons, so it fills that capacity with better-organized structure. The gain is in the organization, not the size.\nVariations on the theme Matching softened output odds is the classic recipe, and the family is broader. Sometimes the student matches not just the teacher\u0026rsquo;s final odds but the intermediate values flowing through the middle of the teacher\u0026rsquo;s tower, so it reproduces inner workings rather than conclusions. Sometimes there is no separate teacher and student at all: a large model generates a body of high-quality answers and a smaller model trains on that text in the ordinary way. That looser form has become the common one for building lean open models, and it is worth noticing what it gives up — training on generated text sees only the words the teacher happened to emit, not the distribution behind them, which is the entire quantity the classic recipe was built to transfer. Distillation also pairs with the other shrinking techniques, trimming unneeded knobs or storing each knob in fewer bits: distillation shrinks what the model knows into fewer knobs, those methods shrink the cost of each knob.\nWhat the student inherits The transfer is faithful, and that is the cost as well as the mechanism. Mass-covering KL asks the student to reproduce the teacher\u0026rsquo;s distribution rather than the truth, so a teacher\u0026rsquo;s confident errors arrive intact and are learned as carefully as its correct rankings; nothing in the loss distinguishes them. The teacher is also a ceiling. A student trained to match a distribution cannot be pushed past it by more training on the same teacher, which is why gains from distillation stop where the teacher\u0026rsquo;s competence stops. And the transferred skill is not the capability that produced it. Where a reasoning model found its habits through an expensive search against a verifier, as the article on RLVR describes, a distilled student imitates the traces and never runs the search — it arrives holding the behaviour without the means of having discovered it. The saving is real and it is a saving on inference, not on the teacher: somebody paid to train the large model, and somebody paid to run it across the whole distillation set.\nFigure 3. What crosses over with the competence.\nWhat to carry forward Distillation transfers a large model\u0026rsquo;s judgement into a small one by training the student on the teacher\u0026rsquo;s soft odds rather than on hard, single-word answers. The teacher\u0026rsquo;s graded uncertainty — how it ranks every wrong answer, not merely which one is right — is the lesson ordinary training throws away. A temperature dial brings that texture into view, the KL divergence measures the student\u0026rsquo;s imitation, and a mixing weight $\\alpha$ blends the soft lesson with the ordinary hard one, with a factor of $T^2$ restoring the gradient the softening removed. The payoff is a model carrying much of a giant\u0026rsquo;s competence at a fraction of its running cost, and it is why so many small, fast models are trained against large ones rather than from scratch. What travels with the competence is everything else in the teacher\u0026rsquo;s distribution: its errors, its ceiling, and none of the search that found the behaviour in the first place.\nThe figures in this article follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/articles/distillation/","summary":"\u003cp\u003eThe machinery that makes a large model good is well enough known: attention letting every word survey every other, a tall tower of blocks, and a long, expensive training run that settles billions of knobs into a capable configuration. Capability comes at a price. The best models are enormous, and every answer burns memory and time in proportion to their size — tolerable in a demonstration, punitive across billions of requests. So the wish is obvious: keep most of what a giant model knows while paying for a small one. Distillation is the technique that grants it, by transferring the large model\u0026rsquo;s judgement into a smaller one. This piece is about how the transfer works, why it works better than the size difference suggests, and what the student inherits along with the skill.\u003c/p\u003e","title":"Distillation — Teaching a Small Model to Think Like a Big One"},{"content":"There is a boundary running through post-training. Where an answer can be checked by a program, verifiable rewards give a training signal that cannot be flattered, and we can push on it as hard as we like. But most of what we ask a model to do has no checker. There is no program that decides whether an explanation was pitched correctly, whether a refusal was gracious, whether a summary kept the right things. For that territory — which is most of the territory — we are back to human preference, and the standard way of learning from it is a three-stage apparatus with a reinforcement learning loop at the end. Direct preference optimization is the discovery that the whole apparatus collapses into a single loss you can train the way you train anything else. This piece is about how something that looks like reinforcement learning turns out, on inspection, to be classification.\nThe three-stage machine, and what each stage costs The conventional pipeline runs like this. First, fine-tune the base model on curated demonstrations, giving something that answers questions rather than continuing them. Second, collect pairs of responses to the same prompt, ask people which they prefer, and train a separate network — the reward model — to predict those judgements. Third, run reinforcement learning: let the model generate, score each generation with the reward model, and update toward higher scores, with a penalty on how far the model\u0026rsquo;s output distribution is allowed to drift from where it started.\nIt works, and it produced every assistant you have used. But look at what the third stage requires resident in memory at once. The model being trained. A frozen copy of where it started, needed to compute the drift penalty. The reward model, doing the scoring. And, if you are using the standard algorithm , a critic network estimating how good a partial generation is likely to turn out. Four models, of which two or three are the size of the one you actually want. Inside the training loop there is a sampling loop, because the model must generate its own material to be scored on, and generation is slow. The whole arrangement has a reputation for instability and carries a long tail of hyperparameters that interact with each other in ways nobody finds intuitive.\nAnd hanging over it is the problem that dogs every reward model: it is a proxy, it can be satisfied without the underlying preference being satisfied, and the drift penalty exists precisely to stop the model from finding the places where the proxy is most wrong. The leash is not a refinement. It is load-bearing.\nFigure 1. Four models and a sampling loop, against one loss.\nWhat is the reward model actually for? Step back and notice the shape of the pipeline, because the question almost asks itself.\nWe start with data: a pile of triples, each one a prompt, a response people preferred, and a response they did not. We finish with a policy: a model that behaves the way those people wanted. In between we build a reward model, which is a compression of the preference data into a network, and then we spend an expensive reinforcement learning phase querying that network to recover what we compressed into it.\nThe reward model, in other words, is an intermediate. Nothing about the final product requires that it exist — it exists because reinforcement learning wants a reward function, and preferences are not a reward function. It is a translation step, performed so that the next stage can consume its output, and like any translation it loses something on the way through.\nSo the question is whether the translation can be skipped: whether there is a path from preference pairs directly to a policy, with nothing in between. There is, and the route to it runs through a piece of algebra that has been sitting in plain sight since the objective was first written down.\nA pinch of math: the reward hiding inside the policy The thing reinforcement learning from human feedback is trying to maximize is this: high reward on average, minus a penalty proportional to how far the policy $\\pi$ has strayed from the reference $\\pi_{\\text{ref}}$ it started at, with $\\beta$ setting how tight the leash is.\nIt is a standard result — and the derivation is short, though we will not do it here — that this objective has an exact optimum, and that the optimum can be written down in closed form:\n$$\\pi^*(y \\mid x) = \\frac{1}{Z(x)}\\, \\pi_{\\text{ref}}(y \\mid x)\\, e^{\\,r(x,y)/\\beta}$$Read it as a statement about reweighting. The best possible policy is the model you started with, with every response\u0026rsquo;s probability multiplied by the exponential of its reward, then renormalized so the whole thing still sums to one. High-reward responses are made more likely in proportion to how good they are; the reference distribution supplies everything else. The $Z(x)$ is that renormalization, and it is hopeless to compute, since it requires summing over every response that could ever be written.\nHere is the move. That equation relates a reward to a policy — so turn it around and solve for the reward instead:\n$$r(x,y) = \\beta \\log \\frac{\\pi^*(y \\mid x)}{\\pi_{\\text{ref}}(y \\mid x)} + \\beta \\log Z(x)$$Which says something quite striking. Any policy is implicitly the optimal policy for some reward function, and that reward function can be read directly off the policy: it is how much more likely this policy makes a response than the reference did. No separate network is needed to hold a reward, because a model that has a reference to be compared against already is one.\nThat is the insight, and there remains one obstacle: the intractable $Z(x)$ is still sitting there. It disappears for a reason worth savouring. The standard model of how a human converts two options into a preference — the Bradley–Terry model, borrowed from the statistics of paired comparisons — says the probability of preferring $y_w$ over $y_l$ depends on the difference of their rewards, passed through a logistic curve. Both responses answer the same prompt $x$, so both carry the same $\\beta \\log Z(x)$ term, and in a difference it cancels exactly.\nWhat is left, after substituting the expression above into Bradley–Terry and taking the negative logarithm, is the entire method:\n$$\\mathcal{L} = -\\log \\sigma\\!\\left( \\beta \\log \\frac{\\pi_\\theta(y_w \\mid x)}{\\pi_{\\text{ref}}(y_w \\mid x)} \\;-\\; \\beta \\log \\frac{\\pi_\\theta(y_l \\mid x)}{\\pi_{\\text{ref}}(y_l \\mid x)} \\right)$$You do not need to have followed every step; the thing to carry away is what the final line contains and what it does not. It contains the model being trained, a frozen reference, a preferred response, a rejected one, and a single dial. There is no reward model anywhere in it, no sampling, no critic, no reinforcement learning of any kind. It is a logistic loss over pairs — the same object you would write down for an ordinary binary classifier — and it is trained by feeding it a fixed dataset, exactly like supervised fine-tuning.\nThere is a pleasing echo here of something from the article on logits . A single log-probability, quoted alone, is not a reward and cannot be one; there is no fixed zero on that scale. But a ratio against a reference is perfectly well defined, in the way that an altitude is meaningful once someone agrees on sea level. DPO\u0026rsquo;s reward is a measurement taken against a benchmark, and the benchmark is the model you began with.\nWhat the loss is actually doing The gradient is worth a paragraph, because it makes the behaviour concrete.\nDifferentiating that loss gives an update that raises the log-probability of the preferred response and lowers the log-probability of the rejected one — no surprise — multiplied by a coefficient that is large exactly when the model\u0026rsquo;s implicit reward currently has the pair ordered backwards. A pair the model already scores correctly, by a comfortable margin, produces a coefficient near zero and contributes almost nothing to the update. A pair it gets confidently wrong dominates.\nSo the training automatically concentrates on its own mistakes, which is a property the reinforcement learning version has to work harder for. And the reference model earns its place twice over: it defines the scale on which the implicit reward is measured, and by appearing in both log-ratios it keeps the policy anchored. The leash is not bolted on as a separate penalty term here. It is structural — a consequence of the reward being defined as a ratio to the starting point in the first place.\nThe dial $\\beta$ has the same meaning it had in the original objective. Small values loosen the leash and let the model move a long way to satisfy the preferences; large values keep it close to where it started. It is the one hyperparameter that genuinely matters, which after the tuning burden of the three-stage pipeline feels like a gift.\nFigure 2. The gradient concentrates on pairs ordered backwards.\nTwo things the shortcut costs you It would be dishonest to present this as free, and the two costs are both real.\nThe first is that DPO is off-policy. The preferred and rejected responses in the dataset were written by someone or generated by some other model, at some point in the past. Reinforcement learning, whatever its expense, has the model generate its own candidates and learn from feedback on those — it is always learning about the region of response space it actually occupies. DPO instead learns from a fixed pile of examples that may be nothing like what the current model would produce, and it is being asked to lower the probability of rejected responses it would never have generated anyway. As training proceeds and the policy moves, the gap between the dataset\u0026rsquo;s distribution and the model\u0026rsquo;s own widens, and the signal becomes correspondingly less relevant. This is the main theoretical objection, and it is why the supervised fine-tuning phase beforehand matters more than it looks: the closer the reference model is to the distribution the preference data was drawn from, the less severe the mismatch.\nThe second is stranger, and was noticed only after the method was in wide use. In practice the log-probability of the preferred response usually goes down during DPO training. Not just the rejected one — both. What rises is only the margin between them, which is all the loss ever asked for. The loss is perfectly satisfied by making the rejected response much less likely and the preferred response merely somewhat less likely, and gradient descent, being indifferent to our intentions, frequently takes that route. The probability mass has to go somewhere, and it flows to responses that appear nowhere in the training pair — sometimes harmless, sometimes not. This is likelihood displacement, and it is the mechanism behind a family of odd failures: models that become terser and terser, or that drift toward a bland register that was never anybody\u0026rsquo;s preferred answer, or that in the worst reported cases shift mass from a preferred refusal onto a compliance nobody wanted. The margin was the wrong thing to have optimized alone, and nothing in the objective says otherwise.\nFigure 3. Likelihood displacement: both probabilities fall.\nThe family that grew around it Both problems have attracted fixes, and the result is a small ecosystem rather than a single method.\nSeveral variants change the objective\u0026rsquo;s shape. One line of work observes that when preferences in the data are effectively deterministic — every annotator would agree — the logistic loss can be driven toward an unbounded margin, overfitting badly, and replaces it with a bounded alternative that cannot run away. Another abandons pairs entirely and learns from responses individually labelled good or bad, which is a far easier thing to collect at scale than matched comparisons. Others remove the reference model altogether, either by folding the preference term into the supervised fine-tuning loss so the two happen at once, or by using a length-normalized average log-probability as the implicit reward directly, which has the side benefit of attacking the verbosity bias that afflicts every preference method.\nThe most conceptually direct fix is to make the method less off-policy: sample fresh responses from the current model, have them ranked — by people, or increasingly by a stronger model acting as judge — and run DPO on those, then repeat. Iterating in this way recovers much of what on-policy training provides while keeping each round cheap and stable. It has become the default recipe for open-weight post-training, and it is a fair description of what most people now mean when they say they are doing DPO.\nWhere all this sits relative to verifiable rewards is worth stating plainly, because the two methods are complements rather than rivals. Verifiable rewards apply where a checker exists and permit very aggressive optimization; preference methods apply everywhere else and are inherently limited by the fidelity of a proxy. A modern pipeline uses both — preference optimization for helpfulness, tone, formatting, and safety, verifiable rewards for mathematics, code, and reasoning — and the reason DPO in particular became ubiquitous is not that it beats a well-tuned reinforcement learning setup on quality. It is that it removes three models from memory and most of the hyperparameters from the problem, which put a capability that had belonged to a handful of large labs within reach of anyone with a preference dataset and a single machine.\nWhat to carry forward DPO replaces the reward-model-plus-reinforcement-learning half of preference training with one supervised loss. The step that makes it possible is the observation that the usual objective — maximize reward, stay near a reference — has a closed-form optimum, which can be inverted to show that any policy already is a reward model: its implicit reward for a response is $\\beta$ times the log of how much more likely it makes that response than the reference does. Feed that expression into the Bradley–Terry model of preference and the intractable normalizing term cancels, because a preference depends only on the difference between two rewards for the same prompt, leaving a plain logistic loss over chosen and rejected pairs with no reward model, no critic, no sampling loop, and one dial. Its gradient weights each pair by how badly the model currently has it ordered backwards, so training concentrates on its own errors, and the leash to the reference is structural rather than an added penalty. The costs are that it is off-policy, learning from a fixed dataset that drifts further from the model\u0026rsquo;s own behaviour as training proceeds, and that it optimizes only the margin — so the preferred response\u0026rsquo;s probability commonly falls too, with the displaced mass landing on text nobody chose. The variants that followed either bound the objective, drop the pairing requirement, remove the reference model, or restore on-policy sampling by regenerating and relabelling each round; that last recipe is now the standard. And the whole family occupies the ground that verifiable rewards cannot reach — everything about a model\u0026rsquo;s behaviour that no program can check.\nThe figures in this article follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/articles/dpo/","summary":"\u003cp\u003eThere is a boundary running through post-training. Where an answer can be checked by a program, \u003ca href=\"/articles/rlvr/\"\u003everifiable rewards\u003c/a\u003e\n give a training signal that cannot be flattered, and we can push on it as hard as we like. But most of what we ask a model to do has no checker. There is no program that decides whether an explanation was pitched correctly, whether a refusal was gracious, whether a summary kept the right things. For that territory — which is most of the territory — we are back to human preference, and the standard way of learning from it is a three-stage apparatus with a reinforcement learning loop at the end. \u003cem\u003eDirect preference optimization\u003c/em\u003e is the discovery that the whole apparatus collapses into a single loss you can train the way you train anything else. This piece is about how something that looks like reinforcement learning turns out, on inspection, to be classification.\u003c/p\u003e","title":"DPO — Reading the Reward Off the Model Itself"},{"content":"A model does not see words. Long before the first block of the tower does anything, each token has been replaced by a list of a few thousand numbers pulled from a table with one row per entry in the vocabulary. That table is the embedding, and it is the model\u0026rsquo;s only point of contact with language as we write it. What is easy to miss is that nobody wrote the table — it begins as noise and is learned by exactly the same downhill walk that trains everything else, with one peculiarity in how the gradient reaches it. What is easier still to miss is that most models keep two such tables rather than one. This piece is about how a word finds its place, and about why reading a word and reaching for a word turn out to be different skills.\nThe table that begins as noise The embedding table has one row per vocabulary entry and one column per dimension of the model\u0026rsquo;s internal arrow. With a vocabulary of a hundred and twenty thousand tokens and a width of four thousand, that is close to half a billion numbers, all of which start out as small random draws.\nIt is worth dwelling on how empty that starting point is. In a space of four thousand dimensions, random vectors are very nearly perpendicular to one another — not as a curiosity but as an overwhelming statistical fact, since there is so much room that two arrows drawn at random have almost no reason to share a direction. So at initialization the model\u0026rsquo;s position is not that it has bad ideas about which words are related. It is that it has no ideas at all: every token is as unrelated to every other as it is possible to be. There is no seeded knowledge that \u0026quot; cat\u0026quot; and \u0026quot; dog\u0026quot; belong together, no grammatical categories, no sense that \u0026quot; running\u0026quot; is a form of \u0026quot; run\u0026quot;. A linguist could supply all of that in an afternoon, and nobody does.\nEverything the table eventually contains is a residue of a single pressure. The tower downstream has to predict the next token, and it is easier for the tower if words that need similar treatment arrive looking similar. Nothing ever instructs the table to group synonyms. Grouping emerges because a row that behaves like its neighbours lets the machinery above it be reused, and reuse is what a limited number of knobs rewards.\nA lookup is a multiplication in disguise Selecting row $t$ from a table looks like an operation of a different kind from anything else in the network — an indexing step, a database query, something outside the arithmetic. It is not, and seeing why explains almost everything that follows.\nWrite the token as a one-hot vector: a list as long as the vocabulary, zero everywhere except a single one at position $t$. Multiply that vector by the embedding matrix and what comes out is precisely row $t$, because every other row has been multiplied by zero. The lookup is a matrix multiplication whose left operand happens to be mostly zeros.\nNo implementation actually performs that multiplication — gathering the row directly is a hundred thousand times cheaper, and every framework does exactly that. But the equivalence is the conceptual point. It means the embedding table is not a special structure requiring special handling; it is an ordinary weight matrix sitting at the bottom of the network, and backpropagation reaches it by the ordinary route with no exception carved out for it. It also means the peculiar shape of its gradient is not a design decision anyone made. It is a consequence of all those zeros.\nFigure 1. Why the input gradient reaches one row only.\nThe gradient that only touches what it saw When backpropagation runs, a message travels down through every block of the tower and arrives at the very bottom, at each position, in the form of a vector saying: the input here should have been a little more like this and a little less like that. That message is the gradient with respect to the embedded vector at that position.\nNow push it one step further back, through the one-hot multiplication, to the table itself. Because the one-hot is zero everywhere but one place, the gradient with respect to the table is zero everywhere but one row. The message updates the row for the token that was actually there, and no other row learns anything from that position.\nAggregate this across a batch and the consequence is stark. Each row\u0026rsquo;s gradient is the sum of the messages from every position where that token appeared — and exactly zero if it appeared nowhere. The token \u0026quot; the\u0026quot; turns up thousands of times in a single batch, so its row collects thousands of messages and is revised on every step of training, from the first to the last. A token that appears once in ten thousand documents collects nothing for thousands of consecutive steps, then receives a single modest message, then falls silent again. Between the busiest and the quietest rows in the table the ratio of update opportunities can run to a million to one.\nThis is the cleanest example anywhere in a language model of why one global step size cannot work, and it is precisely the situation that per-knob pacing was built for. A rare row\u0026rsquo;s roughness meter has decayed to almost nothing during its long silence, so when its gradient finally arrives the step it takes is long — the optimizer treats a rare event as informative, which it is.\nThere is a subtler consequence of the same silence, and it is one people are frequently surprised by. Decoupled weight decay is applied on every step to every parameter, whether or not a gradient arrived. So a rare token\u0026rsquo;s row is being pulled toward zero on all those thousands of steps when the token is absent, with nothing pushing back. A frequent token\u0026rsquo;s row is shrunk just as often but is constantly re-inflated by its gradients. Some of the well-known relationship between a token\u0026rsquo;s frequency and the size of its embedding is not semantics at all; it is a bookkeeping artifact of decaying rows that nobody was updating. Implementations that apply decay only to the rows a batch actually touched behave differently, which is one reason embedding tables are often given their own optimizer settings.\nTwo maps: reading a word and reaching for one Everything so far concerns the table at the bottom. There is another at the top.\nRecall from the discussion of logits that the final arrow out of the tower is turned into scores by dotting it against a direction belonging to each vocabulary entry. Those directions form a matrix of exactly the same shape as the embedding table — one row per token, one column per dimension. It is usually called the unembedding, and by default it is a completely separate set of numbers, learned independently.\nSo the model keeps two maps of the same vocabulary, and they answer different questions. The input map answers: when I read this token, what should I contribute to the running state? The output map answers: what must the running state look like for me to say this token? Those are not the same question, and the asymmetry is the same one that separates recognition from production in a person. You can read a great many words you would never reach for; you can recognize a face you could not draw. Reading and producing draw on different information about the same word.\nMake it concrete with a word like \u0026quot; the\u0026quot;. As an input, what matters about it is almost entirely structural — a determiner has arrived, a noun is probably coming, and the word itself carries close to no meaning. As an output, what matters is something else entirely: what does a state look like just before a determiner is due? The input map needs to encode what the word contributes; the output map needs to encode what predicts it. There is no reason for those to be the same list of numbers, and for function words they are conspicuously not.\nThe two maps also live in completely different gradient regimes, which is the practical half of the distinction. The input map, as we have seen, is updated sparsely — only the rows that showed up. The output map is updated densely: because the softmax normalizes over the whole vocabulary, every single row participates in every single prediction, and therefore every row receives a gradient at every position. A hundred thousand small corrections per token, against a handful of large ones at the bottom. For a rare word, the input row may go untouched for a week of training while its output row is being nudged several trillion times.\nFigure 2. Two maps of one vocabulary, opposite gradient regimes.\nA pinch of math: the two gradients The input side is the simpler of the two. Write $e_t$ for the row belonging to token $t$. Backpropagation delivers, at each position, a vector saying how the input there should have differed. The gradient for $e_t$ is the sum of those vectors across every position in the batch where token $t$ appeared, and nothing at all if it appeared nowhere. That is the whole story, and its shape is the point: a sum over occurrences, and a hard zero otherwise.\nThe output side is more interesting. The logit for entry $i$ is $z_i = h \\cdot w_i$, where $h$ is the arrow the tower produced and $w_i$ is that entry\u0026rsquo;s row in the output map, and the loss is the negative logarithm of the probability assigned to the token that actually came next. Working the softmax and the logarithm through together — a derivation that simplifies far more than it has any right to — leaves something remarkably clean:\n$$\\frac{\\partial L}{\\partial w_i} = (p_i - y_i)\\, h$$Here $p_i$ is the probability the model assigned to entry $i$, and $y_i$ is one for the correct token and zero for everything else. You do not need to trace the algebra; the instruction it encodes is the entire training signal for the output map in a single line, and it is worth reading slowly. For the correct token, $p_c - 1$ is negative, so the step moves $w_c$ toward $h$ — the next time this state arises, cast a longer shadow. For every other entry, $p_i$ is positive, so $w_i$ is pushed away from $h$, in exact proportion to how much probability it was wrongly given. The word that nearly won is shoved hard. A word the model already considered impossible is barely moved at all.\nTwo things follow. The first is the density noted above: every row is touched at every position, and the strength of the touch is that row\u0026rsquo;s own probability, which is why the output map is shaped mostly by near-misses. The second is that the same quantity, gathered back across the whole vocabulary, is what flows down into the tower:\n$$\\frac{\\partial L}{\\partial h} = \\sum_i (p_i - y_i)\\, w_i$$Read as an instruction to the layer below: point more like the word you should have said, and less like each word you were tempted by, weighted by how tempted you were. Every gradient that trains the entire tower begins as this one vector.\nFigure 3. The push and pull of the output gradient.\nTying the two together Given two matrices of identical shape holding related information, an obvious economy suggests itself: use one matrix for both. This is weight tying, and for years it was close to universal.\nThe argument for it was partly principled and partly financial. Principled, in that a word\u0026rsquo;s input and output representations ought to have something to do with each other, and forcing them to coincide is a form of regularization that stops the two from drifting into unrelated encodings of the same thing. Financial, in that when models were smaller the embedding was an enormous share of the total: at half a billion parameters, two untied maps could easily be most of a one-billion-parameter model, and halving that was not a marginal saving. Tying also means a rare token\u0026rsquo;s row inherits the dense output gradient, so it is being shaped continuously even during the long stretches when the token itself never appears — which, for the tail of the vocabulary, is most of training.\nThe argument against it is the one made above: the two jobs are different, and a single set of numbers cannot be optimal for both. A tied row is a compromise, and it is not an even-handed one. The output gradient arrives at every step and the input gradient arrives occasionally, so the compromise is settled overwhelmingly in favour of production. The row ends up being mostly an answer to \u0026ldquo;what precedes this word,\u0026rdquo; with the reading role fitted around it.\nWhich consideration wins has shifted with scale. When the vocabulary is large relative to the model, tying is close to free and the regularization is welcome. As models grew, the embedding became a small fraction of the whole, the saving stopped mattering, and the constraint started to cost measurable quality — so the largest models today generally keep the two maps separate, while smaller ones and those with unusually large vocabularies often still tie them. It is a genuine trade rather than a solved question, which is why you will find both in current systems.\nWhat the map actually looks like It is worth being straight about how much of the popular picture survives contact with a real table, because the folk version is not quite wrong but is oversold.\nThe famous demonstration — that the arrow from \u0026quot; man\u0026quot; to \u0026quot; woman\u0026quot; is roughly the arrow from \u0026quot; king\u0026quot; to \u0026quot; queen\u0026quot;, so that arithmetic on vectors performs analogy — comes from an earlier generation of static embeddings, models like word2vec and GloVe whose entire job was to produce one good vector per word. There, the vector had to be the final word on meaning, and the training objective pushed hard to make the geometry regular. In a transformer\u0026rsquo;s input table the effect is present but weaker and considerably less reliable, and the reason is structural: the input embedding is not the final word on anything. It is a starting guess that thirty or eighty blocks of attention will refine using context, and a great deal of what we mean by a word\u0026rsquo;s meaning is assembled up there rather than looked up down here. The input row for \u0026quot; bank\u0026quot; does not have to resolve which sense is intended. It only has to give the tower enough to work with.\nTwo other features of real tables are worth knowing. Learned embeddings do not spread out to fill the space available to them; they occupy a surprisingly narrow cone, so that two arbitrary tokens have a much higher cosine similarity than intuition suggests. Anyone comparing embeddings directly has to centre or whiten them first, or the similarity scores will be dominated by the shared direction that every token has in common. And frequency leaves a heavy imprint on the geometry — often the single largest axis of variation across the table tracks how common a token is rather than anything you would call semantic, which is a useful thing to remember before reading meaning into a scatter plot.\nFinally, the map settles early. Most of the structure in the embedding table is in place within the first small fraction of training, after which the tower above continues to improve for a long time while the table beneath it changes comparatively little. This is a large part of why freezing the embeddings during fine-tuning is usually free, and why so many adaptation methods leave them alone entirely: the map is not where the remaining work is.\nWhat to carry forward An embedding table is an ordinary weight matrix with one row per vocabulary entry, initialized as noise and learned by the same descent as everything else. Because a lookup is really a multiplication by a one-hot vector, the gradient reaching it is zero for every row the batch did not contain and, for the rows it did, the sum of the messages arriving from each occurrence — so a row\u0026rsquo;s update frequency tracks its token\u0026rsquo;s frequency, across a range that can span a million to one. That sparsity is the textbook case for per-knob step sizes, and it also means decoupled weight decay quietly shrinks a rare token\u0026rsquo;s row throughout its long silences, which accounts for part of the observed link between frequency and embedding size. Most models keep two such maps: an input embedding answering what a token contributes when read, and an unembedding answering what a state must look like for that token to be said. They are different questions, and they are trained under opposite regimes — the input map sparsely, the output map densely, since the softmax makes every row participate in every prediction and pushes each away from the current state in proportion to the probability it was wrongly given. Weight tying merges the two into one matrix, buying parameters and regularization at the cost of forcing one set of numbers to serve both roles, a compromise decided largely by the dense output gradient; it made obvious sense when the embedding dominated the parameter count and is increasingly abandoned as models grow. And the geometry deserves a colder eye than it usually gets: the analogy arithmetic belongs mostly to an earlier era of static embeddings, real tables sit in a narrow cone that inflates every similarity score, frequency rather than meaning is often the largest axis of variation, and the whole map is substantially finished early in training — which is why leaving it frozen during fine-tuning generally costs nothing at all.\nThe figures in this article follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/articles/embeddings/","summary":"\u003cp\u003eA model does not see words. Long before the first block of the tower does anything, each token has been replaced by a list of a few thousand numbers pulled from a table with one row per entry in the vocabulary. That table is the \u003cem\u003eembedding\u003c/em\u003e, and it is the model\u0026rsquo;s only point of contact with language as we write it. What is easy to miss is that nobody wrote the table — it begins as noise and is learned by exactly the same downhill walk that trains everything else, with one peculiarity in how the gradient reaches it. What is easier still to miss is that most models keep two such tables rather than one. This piece is about how a word finds its place, and about why reading a word and reaching for a word turn out to be different skills.\u003c/p\u003e","title":"Embeddings — The Two Maps a Model Keeps of Its Vocabulary"},{"content":"For a few years the standard illustration of what scale buys you was a plot in which a line lies flat along the bottom of the chart for three or four orders of magnitude and then, past some particular model size, turns upward and climbs almost vertically. The claim attached to it was that the ability being measured is simply absent below that point and present above it — that something switched on. The plot did an enormous amount of work. It appeared in talks about safety, in arguments about regulation, and in forecasts of what the next training run would do. It is also, for a large fraction of the abilities it was drawn for, a property of how the vertical axis was defined rather than of the models along the horizontal one. That does not settle the matter, because a residue survives the objection and the residue is the interesting part. This piece is about what the original claim asserted, the deflationary account that dismantles most of it, and what is left standing.\nWhat was claimed, and why it mattered The survey that named the phenomenon, from 2022, ran a large collection of benchmark tasks against model scale and picked out a subset with a characteristic shape: performance indistinguishable from chance across every smaller model, then a sharp rise somewhere. Three-digit arithmetic. Multi-step word problems. The term offered for this was emergent abilities, borrowed loosely from the physical sciences, where it names a property of a system that its parts do not have.\nThree separate assertions ride inside that one word, and they are almost never argued apart. Unpredictability is the claim that you could not have forecast the ability from smaller models. Discontinuity is the claim that it arrives sharply rather than accumulating. Qualitative novelty is the claim that the larger model is doing something different in kind. These come apart easily. A quantity can be continuous and still unpredictable if all you have measured is its floor, and it can jump on a chart without anything new coming into existence. The bundling is not innocent. The force of the term comes from novelty; the evidence offered is nearly always discontinuity on a plot.\nWhat made the claim land hard was timing. Scaling laws had just established that the pretraining loss of a language model is a smooth power-law function of size, data, and compute, straight on logarithmic axes over many orders of magnitude. Emergence said that the quantity anyone actually cares about, which is what the model can do rather than its cross-entropy on held-out text, does not inherit that smoothness. Loss is forecastable, capability is not. That asymmetry was the entire payload.\nThe cliff is manufactured by the scoreboard Here is the objection. It is strong enough that repeating the emergence claim without addressing it is not an argument.\nTwo different quantities are being conflated. The smooth thing is the model\u0026rsquo;s loss, a continuous function of the distribution it puts over the next token. The jagged thing is a benchmark score, which is a function applied on top of that distribution — and the function is almost always a threshold. Exact-match accuracy on a multi-step answer pays nothing until every step is right. It is a step function composed with a smooth one, and such a composition can be as sharp as the step is.\nWork an example through. Take a task needing five arithmetic steps, scored only on the final answer. A small model gets each step right three times in ten; a larger one, six; a larger one still, nine. The underlying competence has tripled, smoothly, in increments you could have plotted. The exact-match scores are roughly two-tenths of a percent, eight percent, and fifty-nine percent. Flat, flat, cliff. Nothing discontinuous happened inside the model. The discontinuity was made by the scoring rule and then attributed to the thing scored.\nThe test is to change the ruler and look again. Score the same models on token edit distance to the correct answer, or on the probability they assign to it — a quantity read straight off the logits — or on partial credit per step. Any measure continuous in the outputs rather than thresholded on them. Do this across the tasks originally offered as examples and the cliffs frequently straighten into the curve the loss was already following; for a large share of them the sharpness does not survive the change of metric. That is close to decisive. A property that vanishes when you swap the ruler was a property of the ruler.\nAnd a floor is not a measurement. The rest of the objection concerns resolution. If a score sits at chance for every model below some size, that stretch of the plot records only that the instrument bottomed out. Absence of evidence is being read as evidence of absence. Give the small models credit for partial progress, or sample them enough times to resolve a success rate of one in ten thousand, and they generally turn out to have been improving all along, beneath the resolution of the chart. The flat region was never flat.\nThe literature was unusually exposed to this. Evaluation of language models runs on thresholded scores — multiple choice marked right or wrong, generation matched against a reference string, code passing or failing a test. A field that measures nearly everything with step functions, then goes looking for steps in its measurements, will find them. The artifact was not a subtle statistical trap. It was the default instrument.\nFigure 1. The same models, two rulers.\nA pinch of math: what a product of probabilities does to a score The product argument is worth writing down, because it yields a prediction rather than an alternative story.\nLet a task require $k$ steps, each of which the model gets right independently with probability $p$. Exact-match accuracy is the product:\n$$A = p^{k}$$Take logarithms and we get\n$$\\log A = k \\log p$$which is linear in $k$. At fixed competence, lengthening a task costs score exponentially. Now let $p$ improve smoothly with scale. For small $k$ the score is as smooth as $p$ is. For large $k$ it stays negligible until $p$ approaches one and then rises steeply, because the sensitivity $dA/dp = k p^{k-1}$ itself vanishes for small $p$ and is largest near $p = 1$. With $k = 10$, moving $p$ from $0.5$ to $0.9$ takes accuracy from about one in a thousand to about one in three. Competence not quite doubled. The score moved by a factor of three hundred.\nThe prediction follows. The apparent threshold sits wherever $p^{k}$ first clears the benchmark\u0026rsquo;s noise floor, and that location depends on $k$. So one model family should show its emergence point at different scales for tasks differing only in length, with longer tasks emerging later. That is what is observed — an odd thing if a capability is switching on inside the network, and exactly what to expect if the sharpness lives in the exponent.\nFigure 2. Where the threshold sits depends on task length.\nWhat survives the deflation The deflationary case is not a proof that behavioural discontinuities cannot happen, and reading it that way is the mirror-image error. It shows that a family of plots does not demonstrate what it was taken to demonstrate. What remains is narrower.\nThe clean cases are mechanistic rather than metric. The best documented is the formation of induction heads : a two-head circuit that finds where the current token occurred earlier in the context and predicts whatever followed it, assembling over a narrow window of a few hundred optimizer steps. Before that window the model cannot exploit repeated structure. After it, it can. The event shows up outside as a small bump in an otherwise smooth loss curve, and the ability to learn from examples inside a prompt appears at the same moment. The kink is in the cross-entropy itself rather than a thresholded score laid on top, which makes this case immune to the metric objection.\nThat is emergence over training time, not over scale. The two are routinely conflated, and the conflation flatters the weaker of them. A discrete mechanism assembling at a particular step in a particular run is well evidenced; a capability appearing between one model size and the next is the claim the deflationary argument mostly dissolves. The training-time case is better documented by a wide margin, and it is not evidence for the scale version.\nThe mechanistic evidence has its own ceiling. The circuit was traced, and switched off by ablation , in small models built for study. At production scale the ablations were never run, so the account there is correlational, and superposition is the reason: components in a large model participate in many circuits at once. Nor is every discontinuity in a loss curve a circuit being born — loss spikes are an optimization pathology with no capability attached, a reminder that a kink is a thing to explain rather than an explanation.\nFigure 3. Three claims bundled inside one word.\nBoth consequences are being acted on at once The disagreement is not academic: the two readings imply opposite practices and both are in use.\nIf it is mostly artifact, forecasting is easier than advertised. Then the right move is to pick metrics continuous in the model\u0026rsquo;s outputs — per-step credit, probability of the correct answer — fit them on small runs, and extrapolate. Capability becomes about as predictable as loss, and the emergence literature was reporting a property of its own scoreboards. Forecasts are now built this way, and it is close to standard practice where planning is required.\nIf any of it is real, the asymmetry bites. No amount of small-scale testing can rule out a behaviour that shows up only later, because the small-scale measurement may be sitting at the floor. The constraint is one-sided: continuous metrics can raise confidence that a capability will arrive, and cannot establish that one will not. This is the honest reason pre-deployment assessment of a larger model is hard, and why safety cases rest on testing the model actually shipping.\nAnd the axis has stopped being one axis. Reasoning models and test-time compute mean a capability can appear because a system was allowed to spend more work per query rather than because it grew, so \u0026ldquo;emerges at scale\u0026rdquo; underspecifies which scale. The word carries three claims and two axes. Splitting it would cost nothing and settle most of the arguing.\nWhat to carry forward The claim was that certain abilities are absent in smaller models and present in larger ones, arriving abruptly at a threshold rather than accumulating, and what mattered was the consequence: capability would not be forecastable even though loss demonstrably is. The strongest objection is that the smooth quantity is the loss while the jagged quantity is a metric laid on top of it, and metrics like exact-match on a multi-step answer pay nothing until every step lands — so a product of smoothly improving per-step probabilities sits near zero and then climbs steeply while nothing inside the model does anything sharp. Swap to a continuous metric and the cliff frequently straightens; the apparent threshold even shifts with task length, exactly as the product account predicts. A second objection compounds it: a score parked at chance is an instrument at its floor, which is not a measurement, and small models given partial credit usually improve all along. What survives rests on different evidence — a specific circuit assembling over a few hundred training steps, visible as a bump in the loss itself rather than in any scoreboard — and that is emergence over training time, not over scale, which is the conflation to watch for. The dispute is unresolved and both readings are acted on: forecasters choose continuous metrics and extrapolate with reasonable success, while anyone responsible for a frontier model treats small-scale testing as unable to establish absence. The asymmetry is the part to keep. Continuous metrics can tell you a capability is coming; nothing available tells you one is not, and a term bundling unpredictability, discontinuity, and novelty into one word makes that gap easier to argue past than to close.\nThe figures in this article follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/articles/emergent-abilities/","summary":"\u003cp\u003eFor a few years the standard illustration of what scale buys you was a plot in which a line lies flat along the bottom of the chart for three or four orders of magnitude and then, past some particular model size, turns upward and climbs almost vertically. The claim attached to it was that the ability being measured is simply absent below that point and present above it — that something switched on. The plot did an enormous amount of work. It appeared in talks about safety, in arguments about regulation, and in forecasts of what the next training run would do. It is also, for a large fraction of the abilities it was drawn for, a property of how the vertical axis was defined rather than of the models along the horizontal one. That does not settle the matter, because a residue survives the objection and the residue is the interesting part. This piece is about what the original claim asserted, the deflationary account that dismantles most of it, and what is left standing.\u003c/p\u003e","title":"Emergent Abilities — When a Curve Looks Like a Cliff"},{"content":"Almost everything else in the machinery of a language model can be verified. A gradient is correct or it is not; a cache holds the right keys or it does not; a quantized model produces outputs close to the original or it does not. Evaluation is the one activity where the ground keeps moving. We build models to be generally capable, then measure them with tests that are necessarily specific; the tests leak into the training data; the numbers rise while the thing being measured may not have improved; and the community moves to a new test and repeats the cycle. None of this is fraud, and most of it is not even carelessness — it is the predictable consequence of optimizing against a measurement. This piece is about why evaluation is genuinely hard, which failures are structural, and what can actually be trusted.\nWhat makes it hard Four difficulties, and they are different in kind rather than degrees of the same problem.\nThe target is unbounded. A model is meant to be good at an open set of tasks that nobody has enumerated. Any benchmark samples from that set, and a sample is only as representative as its designer\u0026rsquo;s imagination. Excellent performance on the sample supports a conclusion about the sample.\nLoss is not capability. As the article on scaling laws noted, the smooth quantity we can measure precisely is cross-entropy on held-out text, and the mapping from that to whether a model can do anything useful is neither smooth nor known.\nMost outputs have no right answer. Multiple choice can be scored. A summary, an explanation, a piece of advice, a paragraph of prose — these have quality rather than correctness, and quality requires a judge.\nThe measurement changes the thing measured. This is the one that dominates everything else. Once a benchmark matters, it is optimized against, directly through training data selection and indirectly through a thousand small decisions. A measure that becomes a target stops measuring what it did.\nContamination This is the most concrete of the problems, and the most widespread.\nModels are trained on text scraped from the internet. Benchmarks are published on the internet, discussed on the internet, reproduced in papers, quoted in blog posts, and included in datasets of collected evaluations. A model that has seen the test items during pretraining is not being tested; it is being asked to recall.\nContamination is rarely deliberate. It arrives through mirrors nobody knew about, through aggregator datasets that silently include benchmark suites, through discussions that quote items verbatim, and through synthetic data generated by a model that was itself contaminated. Deduplication helps and does not solve it, because a paraphrased or translated item does not match by hash while conveying the same answer.\nDetection is imperfect. Comparing a model\u0026rsquo;s perplexity on benchmark items against similar unseen text is suggestive — memorized text is unusually unsurprising — and so is checking whether a model can complete an item from its first few words alone, or whether it performs better on the canonical ordering of multiple-choice options than on a shuffled one. None is conclusive.\nThe strongest defences are procedural rather than statistical. Hold out a private test set and never publish it. Use benchmarks constructed from material published after a model\u0026rsquo;s training cutoff, which is the cleanest possible guarantee and requires continuously generating new ones. Report on tasks whose answers are computed rather than looked up. The general shape of the answer is that a benchmark\u0026rsquo;s useful life is finite from the moment it is published, and that treating benchmarks as durable is the mistake.\nJudging what cannot be scored For open-ended output, someone has to judge. There are three options and each is compromised.\nHumans set the reference standard and charge for it. An annotator costs money, takes days, and disagrees with the annotator beside them: agreement on subjective quality is often mediocre, and human raters have documented biases — toward longer answers, toward confident phrasing, toward fluency over accuracy. The same biases the article on preference learning identified as sources of reward hacking are present in the evaluation, because it is the same signal.\nReference-based automatic metrics compare output against a written answer by string overlap. They cost almost nothing, reproduce exactly, and measure the wrong quantity: a correct answer phrased differently scores badly, and a wrong answer sharing vocabulary scores well. They persist mainly through inertia.\nModel judges are now the dominant approach: present a stronger model with the output and a rubric and ask for a score or a comparison. It costs little, runs fast, and tracks human judgement better than string metrics do. It also carries specific and well-documented biases. Judges prefer longer responses. They prefer the first option presented, so pairwise comparisons must be run in both orders and averaged. They prefer text resembling their own generation style, which means a judge from the same family as one contestant is not neutral. And they are poor at detecting errors they would have made themselves, which puts a ceiling on using a model to evaluate anything near its own competence.\nNone of this makes model judging useless. It makes it an instrument with known distortions, to be calibrated against human judgement on a sample rather than trusted absolutely.\nFigure 1. Three judges, three sets of distortions.\nWhat can actually be trusted Some measurements resist the failures above, and the pattern connecting them is worth extracting.\nExecutable evaluation. Where correctness is checked by running something — code against tests, a proof through a checker, a query against a database — the measure is a fact rather than a judgement. This is the same property that makes verifiable rewards work, and it makes an evaluation similarly hard to game. It does not prevent contamination of the problems, but it does mean partial credit and stylistic flattery cannot be earned.\nHeld-out and freshly-generated tests. Problems built after the training cutoff, or procedurally generated from templates so that no specific instance was ever published, are contamination-proof by construction.\nHead-to-head comparison with real users. Serving two anonymous models to people who came with their own tasks, and collecting preferences, avoids the fixed-question problem entirely and cannot be trained against directly. Its weaknesses are the human biases above and a population that is not representative of any particular use.\nAdversarial and perturbation testing. Rephrase a question, permute the answer options, change the names and numbers. A model that understood the task is robust to these; a model that memorized the item is not. The gap between original and perturbed performance is one of the more honest signals available, and it is reported far too rarely.\nA pinch of math: how many samples the number needs One quantitative point, because benchmark differences are compared far more casually than the sample sizes justify.\nScoring $n$ items with a per-item success probability $p$ gives an observed accuracy whose standard error is\n$$\\text{SE} = \\sqrt{\\frac{p(1-p)}{n}}$$For a benchmark of 500 items at around 70 percent accuracy, this is about $0.020$ — two percentage points. A 95 percent confidence interval spans roughly four points, and comparing two models requires accounting for the uncertainty in both, so a difference smaller than about four or five points on such a benchmark is not distinguishable from noise.\nThis is before considering the other sources of variance, which are usually larger: the choice of prompt template can move a score by several points, the number of examples shown in the prompt matters, the decoding settings matter, and the answer-extraction logic matters. Published tables routinely report differences of one or two points as meaningful. Most of them are not, and the practice of reporting a single number with no interval is the field\u0026rsquo;s most common statistical failing.\nFigure 2. The error bars a benchmark number omits.\nWhat to carry forward Evaluation is hard for reasons that do not go away with effort: the capability being measured is unbounded while any test is a sample of it; the loss we can measure precisely does not map smoothly onto usefulness; most valuable outputs have quality rather than correctness; and any measure that comes to matter is optimized against and stops measuring what it did. Contamination is the concrete face of this — benchmarks are published on the internet and models are trained on the internet, so a test item may be recalled rather than solved, and it arrives through mirrors, aggregators, discussions, and synthetic data rather than through deliberate cheating. Detection is suggestive rather than conclusive, so the real defences are procedural: private held-out sets, problems drawn from material published after the training cutoff, and answers that are computed rather than looked up. For open-ended output a judge is unavoidable, and all three kinds are compromised — humans are inconsistent and biased toward length and confidence, string-overlap metrics measure the wrong thing entirely, and model judges, now dominant, favour long answers, whichever option is presented first, text resembling their own style, and are blind to errors they would have made themselves. What survives is evaluation that is executable, where a program checks the answer; tests generated fresh or held private; head-to-head comparison on tasks real users brought; and perturbation testing, where rephrasing a question or permuting its options separates understanding from memorization. And the numbers deserve error bars: a five-hundred-item benchmark carries a standard error around two points before prompt-format variance is counted, which makes the routine reporting of one- and two-point differences as meaningful the field\u0026rsquo;s most common statistical failing.\nThe figures in this article follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/articles/evaluation/","summary":"\u003cp\u003eAlmost everything else in the machinery of a language model can be verified. A gradient is correct or it is not; a cache holds the right keys or it does not; a quantized model produces outputs close to the original or it does not. Evaluation is the one activity where the ground keeps moving. We build models to be generally capable, then measure them with tests that are necessarily specific; the tests leak into the training data; the numbers rise while the thing being measured may not have improved; and the community moves to a new test and repeats the cycle. None of this is fraud, and most of it is not even carelessness — it is the predictable consequence of optimizing against a measurement. This piece is about why evaluation is genuinely hard, which failures are structural, and what can actually be trusted.\u003c/p\u003e","title":"Evaluation — Measuring Something That Refuses to Hold Still"},{"content":"Every block in a transformer has two halves. Attention gets nearly all the discussion, and the other half — two matrix multiplications with a nonlinearity between them, applied to each token independently — gets described as \u0026ldquo;the feed-forward part\u0026rdquo; and passed over. This is an odd allocation of attention, given that the feed-forward layers hold roughly two thirds of the model\u0026rsquo;s parameters, and given that when researchers go looking for where a model stores the fact that Paris is in France, this is where they find it. Attention moves information between positions. The feed-forward layers are where information is looked up, transformed, and decided upon. This piece is about the half of the architecture that does the knowing.\nWhat it is The operation is the simplest thing in the model. Take the token\u0026rsquo;s vector, multiply by a matrix that expands it to a larger width, apply a nonlinearity, multiply by a matrix that projects it back down:\n$$\\text{FFN}(x) = W_{\\text{down}} \\; \\sigma\\!\\left( W_{\\text{up}} \\, x \\right)$$The expansion is conventionally fourfold — a model of width four thousand expands to sixteen thousand and comes back. Two matrices of four thousand by sixteen thousand is about 134 million parameters per block, against roughly 67 million for the attention projections of the same block. Multiply by eighty blocks and the ratio holds: two thirds of the model, give or take, is this.\nThe crucial structural fact is that this operates on one token at a time. There is no interaction between positions here at all; the same function is applied independently to every token in the sequence, exactly as it would be to a single token in isolation. Whatever a token has gathered from its neighbours, it gathered during the attention step. The feed-forward layer then works on the result privately.\nThat division gives the block a rhythm worth naming. Attention is the part where tokens consult one another; the feed-forward layer is the part where each token, alone, does something with what it heard. Communication, then reflection, repeated eighty times.\nFigure 1. Two thirds of the parameters are not attention.\nWhy the nonlinearity is the whole point Remove $\\sigma$ and the layer collapses. Two matrix multiplications in a row are a single matrix multiplication — $W_{\\text{down}} W_{\\text{up}}$ is just some other matrix — so without the nonlinearity, the expansion and contraction buy nothing whatsoever, and indeed the entire tower would collapse to one linear map no matter how many blocks it had. Everything a network can do beyond drawing a straight line comes from the nonlinear function sitting between its linear ones.\nWhich function has drifted over the years, and the drift is instructive. The original transformer used ReLU, which zeroes anything negative and passes anything positive unchanged — brutally simple, and it works. GELU smoothed the corner, weighting each input by roughly the probability that a standard normal falls below it, which gives a small negative dip just below zero and a differentiable transition. The smoothing helps in practice, though the theoretical case for it has always been thinner than its adoption suggests.\nThe more interesting change was gating. Instead of one expansion, compute two in parallel, pass one through a nonlinearity, and multiply the two together elementwise:\n$$\\text{SwiGLU}(x) = W_{\\text{down}}\\Big( \\sigma(W_{\\text{gate}}\\, x) \\odot (W_{\\text{up}}\\, x) \\Big)$$One branch produces content, the other produces a multiplicative gate deciding how much of that content passes. This is more expressive than a fixed nonlinearity because the gate depends on the input: the layer can learn to suppress a feature in one context and admit it in another, rather than applying the same fixed curve everywhere. It costs a third matrix, so implementations shrink the hidden width — typically to about two-thirds of what it would otherwise be — to keep the parameter count level. Nearly every recent model uses a gated variant, and the empirical gain is consistent, if modest. The original paper introducing them concluded, memorably, that it offered no explanation for why they work beyond divine benevolence, and the situation has not improved much since.\nFigure 2. Gating adds a second branch and a narrower hidden width.\nA pinch of math: the layer as a key-value store Here is the reading that makes the feed-forward layer legible, and it comes from writing the two matrices in terms of their rows and columns rather than as blocks.\nLet $k_i$ be the $i$-th row of $W_{\\text{up}}$ and $v_i$ the $i$-th column of $W_{\\text{down}}$. Then the whole operation can be rewritten as a sum:\n$$\\text{FFN}(x) = \\sum_{i=1}^{d_{\\text{ff}}} \\sigma\\!\\left( k_i \\cdot x \\right) \\, v_i$$Read that carefully, because it is the same shape as attention. Each hidden unit $i$ compares the input against a stored direction $k_i$ by dot product; the nonlinearity turns that comparison into a coefficient; and that coefficient scales a stored output vector $v_i$ which gets added to the total. The layer is a lookup table with sixteen thousand entries, where $k_i$ is what the entry matches on and $v_i$ is what it contributes when matched.\nThe difference from attention is where the keys and values come from. In attention they are computed from other tokens in the sequence, so the lookup is over the present context. Here they are learned weights, fixed after training, so the lookup is over everything the model absorbed from its corpus. Attention retrieves from what is in front of you; the feed-forward layer retrieves from memory.\nThis is not merely a suggestive analogy. Probing the hidden units of trained models finds that individual units respond to interpretable patterns in the input — a unit that fires on text about a particular topic, one that fires at the end of a quoted phrase, one that fires on a specific syntactic construction — and that the corresponding output column, read through the unembedding , promotes a coherent set of tokens. The units really do behave like rules of the form when the input looks like this, push the prediction toward that.\nFigure 3. Attention retrieves from context, FFN from memory.\nEditing a fact The strongest evidence for the memory reading is that you can act on it.\nIf a specific fact is stored as a particular association between a key direction and a value vector in a particular layer, then locating and rewriting that association should change the fact and leave everything else alone. This turns out to be broadly possible. Using the causal tracing methods described under ablation , one can find which layer\u0026rsquo;s feed-forward output is carrying the answer at the moment the subject of a factual statement is being processed, then solve directly for a modification to $W_{\\text{down}}$ that maps the relevant key to a different value. The model then reports the new fact, in varied phrasings, while its behaviour on unrelated inputs is largely unchanged.\nTwo qualifications keep this honest. The edits are less clean than the demonstrations suggest — they often fail to propagate to logical consequences of the edited fact, and applying many in sequence degrades the model. And the localization is real but partial: facts are distributed across several layers and several units rather than sitting in one place, so \u0026ldquo;the layer where this fact lives\u0026rdquo; is an approximation to a smeared reality. Still, the fact that targeted edits work at all is difficult to explain under any account except that these layers are storing associations.\nWhy so wide, and what the width is for The fourfold expansion looks arbitrary and has survived a great deal of scrutiny, which suggests it is near something real. The reason it needs to be wide is that the number of distinct patterns a model wants to recognize vastly exceeds the width of the residual stream it has to read from. Four thousand dimensions cannot hold four thousand independent concepts if the model needs a hundred thousand of them, and the hidden layer is where the stream gets temporarily unpacked into a much larger space where more of those patterns can be separated before being compressed back down.\nThat \u0026ldquo;more patterns than dimensions\u0026rdquo; pressure is a subject in its own right, and the fact that the model resolves it by packing features into near-perpendicular directions rather than dedicated neurons is the reason individual hidden units are usually only partly interpretable. A unit that fires on three unrelated things is not malfunctioning; it is one axis of a space in which the model has stored more directions than it has axes.\nThe mixture-of-experts architecture is best understood as an intervention on exactly this layer. Its \u0026ldquo;experts\u0026rdquo; are feed-forward layers — the attention is left alone — and the observation motivating it is that for any given token, only a small fraction of those sixteen thousand hidden units fire meaningfully. If most of the lookup table is irrelevant to most queries, there is no reason to consult all of it. Splitting the layer into many smaller tables and routing each token to a couple of them keeps the total stored knowledge while consulting only a slice, which is precisely the \u0026ldquo;cheap to compute, expensive to store\u0026rdquo; bargain described in that article. That the sparse-model literature is entirely about this half of the block, and barely touches attention, is a good indication of where the parameters and the knowledge actually are.\nWhat to carry forward The feed-forward half of each block — two matrices with a nonlinearity between them, applied to each token independently — holds roughly two thirds of a model\u0026rsquo;s parameters and is where its learned knowledge sits. Its position in the block gives the architecture its rhythm: attention is where tokens exchange information, and this is where each token, alone, works on what it received. The nonlinearity is not a detail but the entire reason the layer exists, since two consecutive linear maps are one linear map; the field has moved from ReLU through GELU to gated variants that compute a content branch and a multiplicative gate in parallel, which are consistently a little better for reasons nobody has satisfactorily explained. Written out row by row, the layer is a sum over hidden units in which each unit compares the input against a stored key direction and, in proportion to the match, adds a stored value vector — the same shape as attention, except that the keys and values are learned weights rather than other tokens, so the lookup is over what the model absorbed in training rather than over the present context. That reading is supported by probing, which finds units responding to interpretable patterns and promoting coherent sets of tokens, and by the partial success of targeted factual edits that rewrite one key-value association in one layer. The width exists because a model needs to distinguish far more patterns than the residual stream has dimensions, and the observation that only a small fraction of hidden units matter for any given token is exactly what mixture-of-experts exploits — which is why sparse architectures modify this half of the block and leave attention untouched.\nThe figures in this article follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/articles/feed-forward-layers/","summary":"\u003cp\u003eEvery block in a transformer has two halves. \u003ca href=\"/articles/attention/\"\u003eAttention\u003c/a\u003e\n gets nearly all the discussion, and the other half — two matrix multiplications with a nonlinearity between them, applied to each token independently — gets described as \u0026ldquo;the feed-forward part\u0026rdquo; and passed over. This is an odd allocation of attention, given that the feed-forward layers hold roughly two thirds of the model\u0026rsquo;s parameters, and given that when researchers go looking for where a model stores the fact that Paris is in France, this is where they find it. Attention moves information between positions. The feed-forward layers are where information is looked up, transformed, and decided upon. This piece is about the half of the architecture that does the knowing.\u003c/p\u003e","title":"Feed-Forward Layers — Where a Model Keeps What It Knows"},{"content":"Most improvements to attention have been approximations: restrict which positions may see which, or replace the mechanism with something cheaper that behaves roughly the same. This one is not. It computes the identical result to ordinary attention , to the last bit, and it is several times faster and uses a fraction of the memory. That combination should be suspicious — you do not usually get exactness and speed together — and the resolution is that the original implementation was not slow because of the arithmetic it performed. It was slow because of the data it moved. This piece is about the difference between those two things, and about the reordering of the computation that makes the largest intermediate result never need to exist.\nThe bottleneck was never the arithmetic An accelerator has two kinds of memory. There is a large pool, tens of gigabytes, with high bandwidth by any ordinary standard and yet far too slow to keep the arithmetic units fed. And there is a very small pool — tens of kilobytes per processing unit — sitting immediately next to those units, faster by roughly an order of magnitude.\nThe ratio between how fast a chip can compute and how fast it can fetch has been widening for decades. A current accelerator performs several hundred arithmetic operations in the time it takes to bring a single byte from the large pool. Consequently the useful measure of an operation is not how many operations it needs but its arithmetic intensity: operations performed per byte moved. Above a few hundred, the chip is limited by its arithmetic and running near its rated speed. Below, it is limited by memory traffic and idling.\nMatrix multiplication has high intensity, because each element that gets loaded participates in many products. Almost everything else — adding two tensors, applying a nonlinearity, scaling by a constant, taking a softmax — has intensity close to one. Such an operation reads a value, does a trivial amount of work, and writes it back, spending nearly all its time waiting.\nNow look at the standard attention implementation with that lens. It performs a matrix multiply to get the scores, writes the full score matrix to the large pool, reads it back to apply the mask, writes it back, reads it again for the softmax, writes it back, reads it once more to multiply by the values. The two matrix multiplications are efficient; everything between them is memory-bound; and each of those intermediate steps is moving a tensor of size $n^2$ per head.\nThat tensor is the problem. For a sequence of eight thousand tokens with thirty-two heads, the score matrix alone is over four gigabytes in half precision, per layer. Writing it out and reading it back several times is where the time goes.\nFigure 1. Every round trip the standard implementation pays.\nDo not materialize the matrix The idea is to fuse the whole sequence — scores, mask, softmax, weighted sum — into a single pass that never writes the score matrix anywhere, keeping every intermediate in the fast memory beside the arithmetic units.\nThe obstacle is that the fast memory is small. A score matrix of eight thousand by eight thousand does not fit in tens of kilobytes, not remotely. So the computation is tiled: load a block of queries and a block of keys and values, compute the small block of scores those produce, do everything that needs doing with it, accumulate the contribution to the output, discard the block, and move on. The full matrix is computed, piece by piece, and no piece outlives its usefulness.\nThe result is a large reduction in traffic to the slow pool — in practice around ninefold, and provably close to the best any exact method can do for a given size of fast memory. It is worth being precise about what falls and what does not: the memory needed becomes linear in sequence length, because the score matrix is never stored, while the traffic stays quadratic in $n$ and is reduced by a large constant factor rather than to a lower order. The arithmetic is unchanged, and indeed the method does slightly more of it than before. It is faster anyway, because arithmetic was never what it was waiting for.\nFigure 2. Tiling the score matrix that is never stored.\nA pinch of math: making the softmax work in pieces There is a real obstacle to tiling, and overcoming it is the technical heart of the method.\nA softmax is not a local operation. To normalize a row you need its denominator, and the denominator is a sum over the entire row — which means over every key, which is exactly what tiling refuses to have available at once. Processing one block of keys, you cannot yet know what to divide by.\nThe resolution is to compute the softmax incrementally, maintaining a running result and correcting it as each new block arrives. Keep three quantities per query: the largest score seen so far $m$, the running sum of exponentials $\\ell$, and the running weighted sum of values $o$.\nWhen a new block arrives with its own maximum $\\tilde{m}$, the running maximum updates to $m^{\\text{new}} = \\max(m, \\tilde{m})$. Everything accumulated so far was exponentiated relative to the old maximum, so it must be rescaled by the factor $e^{\\,m - m^{\\text{new}}}$:\n$$\\ell^{\\text{new}} = e^{\\,m - m^{\\text{new}}}\\,\\ell \\;+\\; \\sum_{j \\in \\text{block}} e^{\\,s_j - m^{\\text{new}}}$$$$o^{\\text{new}} = e^{\\,m - m^{\\text{new}}}\\,o \\;+\\; \\sum_{j \\in \\text{block}} e^{\\,s_j - m^{\\text{new}}}\\, v_j$$At the end, divide $o$ by $\\ell$ once, and the answer is exactly what the full softmax would have given.\nTwo things are worth noticing. The rescaling factor is always $e$ to a non-positive power, so it is at most one and the arithmetic stays in a safe range — this is the same maximum-subtraction trick that keeps an ordinary softmax stable, described in the article on logits , applied online. And the correction is exact, not approximate. Nothing is being estimated or truncated. That is why the output matches the standard implementation bit for bit, and why this is a change to the schedule of a computation rather than to the computation.\nThe backward pass, and where the memory saving comes from Training needs the gradient, and the gradient with respect to the values requires the attention weights — the very matrix that was never stored. The obvious response is to store it after all, which would surrender the whole gain.\nInstead it is recomputed. During the forward pass, keep only the normalization statistics for each query, which is two numbers per query rather than $n$ — negligible. During the backward pass, recompute each block of scores from the queries and keys that are still available, use the saved statistics to reconstruct the correct softmax values immediately without any second pass, and proceed.\nThis is gradient checkpointing in miniature, and applied here it is a particularly good bargain: the recomputation is a matrix multiply, which is arithmetic-bound and therefore nearly free on hardware that was idling, while the storage it avoids was memory-bound and expensive. The method does more arithmetic and less waiting, which is the right direction on every chip built in the last decade.\nThe memory consequence is the one that changed what was buildable. Attention\u0026rsquo;s memory requirement falls from quadratic in sequence length to linear, which removes the constraint that had made long contexts infeasible regardless of how much time you were willing to spend. It is worth being precise about what this does and does not fix: the time is still quadratic, since every pair of positions is still compared. What disappears is the memory wall. That distinction is why the technique enabled long contexts without making them cheap, and why the KV cache rather than the attention matrix is now what limits how many long conversations can be served at once.\nFigure 3. What falls, what does not.\nThe consequence nobody predicted The most interesting effect of this work was on the research agenda around it.\nBefore it, a substantial literature was devoted to approximating attention — sparse patterns, low-rank factorizations, kernel methods, hashing schemes — all motivated by the quadratic cost, and all trading some accuracy for tractability. Several were genuinely clever. A good many are no longer used.\nThe reason is that they had been competing against a badly implemented baseline. Once exact attention became several times faster and linear in memory, an approximation had to beat that to be worth its accuracy cost, and many could not — partly because the approximations were themselves memory-bound operations with poor arithmetic intensity, and partly because on the sequence lengths people actually use, the quadratic term is not yet dominant. The pressure to approximate was, to a large extent, pressure created by an implementation that moved more data than it needed to.\nThe general lesson is worth extracting, since it recurs. A cost model that counts operations and ignores data movement will systematically misidentify what is expensive, and a research programme built on such a model will optimize the wrong thing. The work that followed — hardware-aware kernels for other operations, fused implementations throughout the stack, the whole practice of writing custom kernels for model internals — descends directly from noticing that once.\nWhat to carry forward FlashAttention computes exactly the same attention as the standard implementation and is several times faster, which is possible because the original was not limited by arithmetic but by data movement: the score matrix, quadratic in sequence length, was written to slow memory and read back for masking, for the softmax, and for the final multiply, and each of those intermediate steps had an arithmetic intensity near one. The method fuses the whole sequence into a single pass that keeps everything in the small fast memory beside the arithmetic units, tiling over blocks of queries and keys so that the full matrix is computed piece by piece and no piece is ever stored. The obstacle is that a softmax needs a sum over an entire row, which tiling withholds — resolved by an online softmax that carries a running maximum, a running denominator, and a running output, rescaling the accumulated totals whenever a larger score appears, which is exact rather than approximate. The backward pass recomputes the score blocks rather than storing them, saving only two statistics per query, which is a good trade because recomputation is arithmetic-bound and storage was memory-bound. Attention\u0026rsquo;s memory cost drops from quadratic to linear while both its time and its slow-memory traffic remain quadratic — the traffic reduced by a large constant factor, around ninefold, rather than to a lower order — so what the technique removed was the memory wall rather than the cost — which is why long contexts became possible and remain expensive, and why the cache rather than the attention matrix is now the binding constraint on serving. And its most lasting effect was to obsolete much of the literature on approximating attention, which had been competing against an implementation that moved far more data than it needed to.\nThe figures in this article follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/articles/flashattention/","summary":"\u003cp\u003eMost improvements to attention have been approximations: restrict which positions may see which, or replace the mechanism with something cheaper that behaves roughly the same. This one is not. It computes the identical result to ordinary \u003ca href=\"/articles/attention/\"\u003eattention\u003c/a\u003e\n, to the last bit, and it is several times faster and uses a fraction of the memory. That combination should be suspicious — you do not usually get exactness and speed together — and the resolution is that the original implementation was not slow because of the arithmetic it performed. It was slow because of the data it moved. This piece is about the difference between those two things, and about the reordering of the computation that makes the largest intermediate result never need to exist.\u003c/p\u003e","title":"FlashAttention — Never Writing Down the Matrix You Do Not Need"},{"content":"The memory accounting in the article on AdamW left something out. Weights, gradients, and optimizer state came to roughly sixteen bytes per parameter, which is a large number and a fixed one — it does not change with how much text you push through the model. Activations are the other half of the story, and they behave completely differently: they scale with the batch size and the sequence length, and for a large model on a long context they can dwarf everything else. The remedy is one of the more satisfying trades in the whole subject, because it exchanges a resource that is hard and expensive to obtain for one that is merely tedious. This piece is about why the backward pass needs so much stored material, and how throwing most of it away and computing it a second time turns out to be cheaper than keeping it.\nWhy the backward pass hoards Backpropagation works by walking the computation graph in reverse, and at every node it needs to know what happened on the way forward.\nThe reason is the chain rule. To find how the loss changes with respect to a matrix $W$ used in $y = Wx$, you need the gradient arriving at $y$ and the input $x$ that was multiplied by it — the derivative is their outer product. The gradient arrives from above during the backward pass. The input $x$ was computed during the forward pass and, unless it was kept, is gone.\nThe same is true nearly everywhere. A nonlinearity\u0026rsquo;s derivative depends on what value passed through it. A softmax \u0026rsquo;s backward pass needs the probabilities it produced. A normalization step needs the statistics it computed. So the standard implementation simply stores everything: every intermediate tensor produced during the forward pass is retained until the backward pass has consumed it.\nThe bill is easy to underestimate. Per layer, per token, a transformer block holds the input to the block, the normalized copy, the query, key and value projections, the attention weights, the attention output, the expanded hidden activation in the feed-forward layer — which alone is four times the model width — and the projection back down. Call it on the order of twenty times the model width per token per layer, in two-byte precision. For a model of width four thousand with eighty layers, processing four thousand tokens, that comes to something in the region of a hundred gigabytes of activations, against a fixed cost of perhaps a hundred and twenty for weights and optimizer state. The activations are not a footnote. They are half the problem, and unlike the fixed half they grow every time you want a longer context or a bigger batch.\nFigure 1. Where a training step\u0026rsquo;s memory goes.\nThe trade Here is the observation the whole technique rests on. The stored activations are not information; they are cached results. Every one of them is a deterministic function of the layer\u0026rsquo;s input, and could be produced again from that input by running the layer forward a second time.\nSo do not store them. Keep only a small number of tensors at chosen points in the network — the checkpoints — and discard everything between them. When the backward pass arrives at a discarded region, take the nearest checkpoint below it, run the forward computation again to regenerate the intermediates for that region, use them, and free them immediately.\nThe cost is one extra forward pass over the recomputed regions. Since a forward pass costs roughly half of a combined forward-and-backward pass, recomputing everything adds about a third to the total compute of a training step — the conventional figure is thirty to forty percent. In exchange, the memory needed for activations falls dramatically.\nThat is a good trade for a reason worth stating plainly: memory is a hard limit and compute is a soft one. Exceeding memory means the run does not start at all. Exceeding a compute budget means the run takes longer. Anything that converts the first kind of problem into the second is worth a great deal, and thirty percent more time is usually a small price for a model that fits.\nA pinch of math: where to put the checkpoints If you checkpoint every layer, memory is minimal and recomputation is maximal. If you checkpoint none, the reverse. The interesting question is what happens in between, and it has a clean answer.\nTake a network of $n$ layers and checkpoint every $k$-th one. Two things then have to be held simultaneously: the checkpoints themselves, of which there are $n/k$, and the activations of the single segment currently being recomputed, of which there are $k$. So the memory required is proportional to\n$$M(k) \\;=\\; \\frac{n}{k} + k$$Minimize over $k$ by taking the derivative and setting it to zero, giving $k = \\sqrt{n}$ and a memory cost proportional to $2\\sqrt{n}$.\nThis is the result that makes the technique remarkable. Storing everything costs memory proportional to $n$; checkpointing at intervals of $\\sqrt{n}$ costs memory proportional to $\\sqrt{n}$, while every activation is computed at most twice. An eighty-layer model checkpointed every nine layers holds roughly eighteen layers\u0026rsquo; worth of activations rather than eighty, for a single extra forward pass. Depth becomes nearly free in memory terms, which is precisely why models could keep getting deeper.\nIn practice the granularity is usually coarser and simpler than the optimum suggests — checkpointing at the boundary of every transformer block, which is $k = 1$ in these terms, because block boundaries are the natural place to cut and because the implementation is a one-line change in most frameworks. The theoretical curve matters more when memory is tight enough to require tuning.\nFigure 2. Memory against checkpoint spacing.\nSelective recomputation The uniform version treats every stored tensor as equally worth discarding, and they are not. Two things vary: how much memory a tensor occupies, and how expensive it is to regenerate.\nThe best candidates are large and cheap. The expanded hidden activation inside a feed-forward layer is four times the model width and is produced by a single matrix multiplication — enormous, trivially recomputed. The output of a normalization step is cheap to redo. Conversely, the attention computation is comparatively expensive to repeat and, with the memory-efficient implementations described under FlashAttention , was never materializing its largest intermediate in the first place.\nSo modern implementations recompute selectively, discarding tensors ranked by memory saved per unit of recomputation cost, and stopping once they fit. The result is typically most of the memory saving of full checkpointing for a fraction of the overhead — single-digit percentages rather than thirty. Some frameworks do this automatically by profiling the graph; others expose it as a policy to be chosen.\nThere is also a variant that changes what the extra resource is. Rather than discarding an activation and recomputing it, offload it to the host machine\u0026rsquo;s ordinary memory and fetch it back when the backward pass needs it. This spends interconnect bandwidth instead of arithmetic, and whether it wins depends entirely on the machine: on a system with a fast link between accelerator and host it can be better than recomputing, and on a system without one it is far worse. The two techniques compose, since a tensor can be offloaded, recomputed, or kept, and the choice can be made per tensor.\nWhere it sits among the other memory techniques It is worth placing this against the neighbouring approaches, because they address different parts of the same bill and are routinely confused.\nLoRA and QLoRA attack the fixed cost — gradients and optimizer state for parameters that no longer need them, and the storage of the frozen weights themselves. They do very little for activations, since a frozen layer still has to be run forward and, if anything downstream of it is being trained, its output still has to reach the backward pass.\nMixed precision attacks both halves proportionally, by making every number smaller.\nParallelism attacks both by dividing them across machines, and one of its variants is aimed squarely at activations for long sequences.\nCheckpointing attacks only the activation half, and it is the only one of the four that spends compute rather than parameters, precision, or hardware. That independence is why it stacks with all of them, and why nearly every large training run has it enabled somewhere in the configuration even when it is not mentioned.\nFigure 3. Four memory techniques, four different currencies.\nWhat to carry forward The backward pass needs, at every node, the values the forward pass produced there — a matrix\u0026rsquo;s gradient depends on the input it multiplied — so the naive implementation stores every intermediate tensor until it has been consumed. That storage scales with batch size and sequence length, unlike the fixed per-parameter cost of weights and optimizer state, and for a large model on a long context it is comparable to or larger than everything else. But those tensors are cached results rather than information: each is a deterministic function of its layer\u0026rsquo;s input and can be regenerated. Gradient checkpointing keeps only a sparse set of intermediate points and recomputes the rest on demand during the backward pass, costing roughly one extra forward pass — thirty to forty percent more compute — in exchange for a large reduction in memory. That is a good trade because memory is a hard limit and compute is a soft one: it converts a run that cannot start into a run that is slower. Checkpointing every $\\sqrt{n}$ layers of an $n$-layer network is optimal, giving memory proportional to $\\sqrt{n}$ with every activation computed at most twice, which is what made great depth affordable; in practice most implementations simply checkpoint at block boundaries. Selective recomputation improves on the uniform version by discarding tensors ranked by memory saved per unit of cost — the fourfold-expanded feed-forward hidden state being the ideal candidate, large and trivially regenerated — recovering most of the saving for single-digit overhead. And the technique is orthogonal to the others, spending compute where LoRA spends trainable parameters, mixed precision spends bits, and parallelism spends machines.\nThe figures in this article follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/articles/gradient-checkpointing/","summary":"\u003cp\u003eThe memory accounting in the article on \u003ca href=\"/articles/adamw/\"\u003eAdamW\u003c/a\u003e\n left something out. Weights, gradients, and optimizer state came to roughly sixteen bytes per parameter, which is a large number and a \u003cem\u003efixed\u003c/em\u003e one — it does not change with how much text you push through the model. Activations are the other half of the story, and they behave completely differently: they scale with the batch size and the sequence length, and for a large model on a long context they can dwarf everything else. The remedy is one of the more satisfying trades in the whole subject, because it exchanges a resource that is hard and expensive to obtain for one that is merely tedious. This piece is about why the backward pass needs so much stored material, and how throwing most of it away and computing it a second time turns out to be cheaper than keeping it.\u003c/p\u003e","title":"Gradient Checkpointing — Trading Recomputation for Room"},{"content":"Training a neural network is routinely described as finding the settings that minimize a loss, which makes it sound like a search — as though a map existed and the procedure consisted of reading it. There is no map. A model in training has access to exactly one thing: the value of the loss at the point it currently occupies, together with the local slope of that loss in every direction at once. It cannot see the next valley, or tell whether the ground ahead rises or falls. From that poverty of information the entire apparatus of training follows — the tiny steps, the schedules, the momentum, the cautious business of nudging a billion numbers a fraction of a percent at a time. What makes any of it affordable is a piece of arithmetic bookkeeping so efficient it is easy to mistake for a triviality. This piece is about the constraint that only local slope is available, the reverse sweep that recovers that slope for every weight at once, and what the resulting number withholds.\nYou cannot see the surface you are standing on Evaluating the loss means running a batch of text through the model and comparing what it predicted against what came next. The result is one number — the cross-entropy , typically between eleven at the start of training and two at the end — covering the current setting of every weight in the tower. It says how wrong the model is here. It says nothing about anywhere else.\nThe temptation is to picture the loss as terrain seen from above, with training picking a route through it. The view is from the ground, in fog, at night. You feel the slope under your feet and that is the whole of it; there is no vantage point and no survey. Every training run ever performed has been conducted under this restriction.\nWhat the slope buys is a guarantee that holds over an unknown distance: move a small enough step against it and the loss will go down. Small enough is doing a great deal of work in that sentence, and nothing in the slope itself says how small. That omission is the origin of learning rates, warm-ups, schedules , and gradient clipping.\nOne number that depends on everything The loss depends on every weight simultaneously, which makes it awkward to picture. A model with a billion parameters defines a function from a billion-dimensional space to one real number, and the object descended is a surface in that space — not a bowl anyone could sketch, a surface whose every point is a complete specification of the model. Geometric intuition trained on three dimensions does not transfer. The algebra does.\nFor each weight, ask a narrow question: holding every other weight fixed, would a slight increase here make the loss rise or fall, and at what rate. That number is a partial derivative. Collect one per weight and the result is the gradient, a vector with one component per parameter, pointing along the steepest increase. Negate it, step a little way, recompute, repeat. This is gradient descent, and it is not one strategy among several. Given only local slope, it is the only move available.\nThere is a second poverty layered on the first. The true loss is defined over the whole corpus, and computing its gradient exactly would mean a forward pass over every document to produce one nudge. Nobody does this. It is estimated instead on a mini-batch of a few hundred thousand tokens — unbiased, but noisy: approximately right and locally wrong. That is why batch size is a quality dial and not merely a throughput one, and why a serious optimizer averages the estimate over time.\nOne sweep instead of a billion There is an obvious way to compute a partial derivative, and it is worth pricing, because the price is the entire point. Nudge one weight by a tiny amount, run the model forward again, observe how much the loss changed, divide by the nudge. This is a finite difference. It requires no calculus and it works.\nIt also requires one full forward pass per weight. For a billion-parameter model that is a billion forward passes to obtain the information for a single step, and a training run needs hundreds of thousands of steps. At any plausible speed the arithmetic runs past the age of the universe. Finite differences are not slow. They are hopeless.\nBackpropagation obtains all billion partial derivatives in one backward sweep costing roughly twice a forward pass. Not a billion sweeps. One. Deep learning rests on that ratio, and the reason it is available is mundane: the chain rule from first-year calculus, applied in reverse to the graph of operations the forward pass just performed, with intermediate results reused instead of recomputed.\nThe reuse is where the saving lives. Every weight in layer forty needs to know how the loss responds to layer forty\u0026rsquo;s output, and that quantity is identical for all of them, so it is computed once. Layer thirty-nine\u0026rsquo;s version of it is obtained from layer forty\u0026rsquo;s by one local multiplication. Walk backwards from the loss to the embeddings and each layer is visited exactly once, handing its sensitivity to its predecessor and pairing it with its own inputs to produce its own weight gradients. Nothing is computed twice.\nThe bill arrives as memory. Producing a layer\u0026rsquo;s weight gradients requires that layer\u0026rsquo;s inputs — the activations from the forward pass — so they cannot be discarded as the forward pass moves on. They must be held until the backward sweep collects them, and the quantity held scales as depth times batch size times width, which for a long sequence exceeds the weights themselves. It is the largest consumer of memory in training, and the reason gradient checkpointing exists to trade recomputation against storage.\nFigure 1. A billion sweeps versus one.\nA pinch of math: the chain rule, run backwards Write the network as a stack of layers, each consuming the previous output along with its own weights:\n$$a_n = f_n(a_{n-1}, W_n)$$with $a_0$ the input, $a_N$ the final activations, and $L$ a scalar computed from $a_N$. Now define one quantity per layer, the sensitivity of the loss to that layer\u0026rsquo;s output:\n$$\\delta_n = \\frac{\\partial L}{\\partial a_n}$$This is the object the sweep carries; everything else is bookkeeping around it. The chain rule gives two statements, and between them they are the whole of backpropagation:\n$$\\delta_{n-1} = \\delta_n \\frac{\\partial a_n}{\\partial a_{n-1}}, \\qquad \\frac{\\partial L}{\\partial W_n} = \\delta_n \\frac{\\partial a_n}{\\partial W_n}$$The first propagates. The factor $\\partial a_n / \\partial a_{n-1}$ is a Jacobian, recording how each output of layer $n$ responds to each of its inputs, and its important property is that it is purely local: it depends on layer $n$\u0026rsquo;s weights and on what layer $n$ saw, and nothing else. No layer needs to know anything about the layers beneath it to hand back their sensitivity.\nThe second extracts. Once $\\delta_n$ is in hand, the gradient for layer $n$\u0026rsquo;s own weights is that same sensitivity paired with the layer\u0026rsquo;s derivative with respect to its weights, which for a matrix multiply is simply the layer\u0026rsquo;s input. Direction from above, inputs from the side, gradients out.\nNow the reason the sweep must run backwards. The recursion builds $\\delta_{n-1}$ out of $\\delta_n$, so it needs a starting value, and only one sensitivity can be written down directly: the one at the top, where $\\partial L / \\partial a_N$ follows from the loss and the logits . Nothing comparable exists at the bottom. A symmetric procedure does run forwards, propagating derivatives of one chosen input rather than of one chosen output, at one sweep per input — a billion sweeps for a billion weights, which is finite differences again. Reverse mode costs one sweep per output, and there is only ever one output: many knobs, one number.\nFigure 2. One sweep down, two statements per layer.\nWhat the gradient does not tell you Four omissions, each with a body of technique attached to it.\nIt gives a direction and never a distance. The gradient\u0026rsquo;s magnitude is in units of loss per unit weight, which is not a step size and cannot be converted into one without curvature the gradient does not carry. Multiplying it by a constant is a guess dressed as a hyperparameter. The whole of AdamW — momentum, per-parameter rescaling, decoupled decay — exists to turn a direction into a defensible step length, and the learning rate still sitting on top of it is set by trial.\nIt contains no curvature. The gradient is a linear approximation to a function that is not linear, valid over a radius it declines to specify. Where the surface bends sharply, a step of the size that worked yesterday overshoots, and the run develops one of the loss spikes that large training jobs are watched for. Second-order methods are well understood; for a billion parameters the object they want is a billion by a billion matrix, which is why they remain an interest rather than a practice.\nIt requires everything to be differentiable. A gradient reaches back only through operations that respond smoothly to their inputs, so any hard decision inside the model is invisible to training. This has shaped the architecture more than it is credited for: a softmax where a maximum would be the honest operation, soft weighted averages inside attention where selection is what is meant. Where a hard decision is genuinely required, the recourse is to pretend in the backward pass that it was something smoother — a working hack, candid about being one.\nIt finds a local minimum and claims nothing about the global one. This is the first objection anyone raises, and it matters less than intuition insists. A point of zero gradient is a minimum only if the surface curves upward in all billion directions; if even one curves down, the point is a saddle and descent can leave along it. A billion conditions holding at once is a strong demand, and almost every critical point here is a saddle. Whether that is because most reachable minima are of comparable quality, or because noisy descent from a sensible initialization never comes near a bad one, is unsettled.\nFigure 3. Four omissions and the machinery each demands.\nWhat to carry forward Training runs under one severe restriction: the loss can be evaluated at the point currently occupied and its local slope measured there, and nothing more. No survey exists, so moving against the slope is not a chosen strategy but the only move on offer. The loss is a scalar function of every weight at once, so the object descended is a surface with as many dimensions as the model has parameters, and the gradient is the vector of partial derivatives pointing along its steepest rise. Recovering it by nudging weights one at a time would cost a forward pass per weight — a billion passes for one step — not expensive but arithmetically hopeless. Backpropagation recovers all of them in one backward sweep costing about twice a forward pass, by applying the chain rule in reverse and reusing each layer\u0026rsquo;s sensitivity rather than recomputing it; that ratio is why deep learning is possible at all, reverse rather than forward because there is one loss and very many weights. The price is memory, since the backward pass needs the forward pass\u0026rsquo;s activations, so storage scales with depth times batch times width — the trade gradient checkpointing exists to make. What emerges is narrower than it looks. It is a direction with no distance attached, which is why an optimizer and a schedule are separate machinery; it carries no curvature, so the safe step size is never known in advance; it requires every operation to be differentiable, which has quietly determined what the architecture may contain; and it converges to a local minimum with no guarantee about any other — a weaker complaint than it sounds, since in a billion dimensions almost every critical point is a saddle, though whether the minima actually reached are comparable in quality is still unsettled.\nThe figures in this article follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/articles/gradient-descent/","summary":"\u003cp\u003eTraining a neural network is routinely described as finding the settings that minimize a loss, which makes it sound like a search — as though a map existed and the procedure consisted of reading it. There is no map. A model in training has access to exactly one thing: the value of the loss at the point it currently occupies, together with the local slope of that loss in every direction at once. It cannot see the next valley, or tell whether the ground ahead rises or falls. From that poverty of information the entire apparatus of training follows — the tiny steps, the schedules, the momentum, the cautious business of nudging a billion numbers a fraction of a percent at a time. What makes any of it affordable is a piece of arithmetic bookkeeping so efficient it is easy to mistake for a triviality. This piece is about the constraint that only local slope is available, the reverse sweep that recovers that slope for every weight at once, and what the resulting number withholds.\u003c/p\u003e","title":"Gradient Descent and Backpropagation — How a Model Learns Which Way to Move"},{"content":"The usual account of attention\u0026rsquo;s cost is that it grows with the square of the sequence length, and the well-known responses are aimed at that square. A machine serving a large model to a few hundred people at once is not usually failing on arithmetic. It is failing on space. Somewhere in its memory sits one key vector and one value vector for every token, of every head, in every layer, for every conversation in flight, and that structure — not the parameter count, not the score grid — decides how many people the machine can hold and therefore what a token costs to sell. The remedy has to be storing less per token. The technique that delivered it rests on an observation close to an insult: most of what is stored is redundant. This piece is about which part of an attention head carries its specialization, what it costs to make heads share the rest, and why the saving turns out to be so nearly free.\nThe bill lands at inference, not at training The quadratic term is a training and prefill cost, since both process whole sequences at once and pay in full for the grid of scores. Generation does not work that way. A decode step handles one new token, whose query attends over everything stored so far, so the grid for that step is a single row. The square has collapsed to a line. The storage has not.\nThe key-value cache holds, for every token already read, one key and one value for each head of each layer, those being what future positions consult. It persists for the life of the request and grows by one entry per generated token. For a large model at a long context it runs to tens or hundreds of gigabytes, more than the weights beside it.\nNow the asymmetry that makes it govern the economics rather than merely appear in them. The weights are read once per forward pass and used by every sequence in the batch, so sixty-four concurrent users amortize one pass over the weight matrices — the whole reason batching works, and why enough arithmetic sits idle for speculative decoding to pay. The cache amortizes over nobody; each sequence reads only its own. The weights are a fixed charge. The cache is per head.\nSo the binding constraint is how many caches fit beside the weights, and the chain from there is short: cache size sets batch size, batch size sets throughput, throughput sets the price. Which makes what follows an inference technique wearing an architecture\u0026rsquo;s clothing. It alters the definition of the model, must be settled before training, and is justified by a figure on an invoice.\nThe queries are where the specialization lives A head, mechanically, is three projections. It takes the token\u0026rsquo;s current vector and produces a query saying what it looks for, a key advertising what it offers, and a value carrying what it hands over; attention runs many heads in parallel on slices of the representation, and they specialize without instruction. One tracks the preceding token, one matches an open bracket to its close, one carries a subject across a clause.\nAsk where in those three the specialization is implemented. The query is the head\u0026rsquo;s question, and a head is defined by what it asks about. The key advertises properties the token possesses — a plural noun, an opened bracket, a concern with cooking — and those properties are nobody\u0026rsquo;s private business. The value is content, and content is content. Sixteen heads asking sixteen questions of the same noticeboard are already sixteen notions of relevance. The board does not need sixteen copies.\nThe claim is checkable rather than decorative. Keys produced by different heads in a layer turn out to be strongly correlated, and so do the values; a head\u0026rsquo;s output degrades far less when forced to read a neighbour\u0026rsquo;s keys than to ask a neighbour\u0026rsquo;s question. Strongly correlated is not identical, and that gap is where the quality loss comes from. The direction lands conveniently, though: the queries that carry the specialization are the one projection never worth storing, since a query is used once and discarded. The expensive part of a head is the part that matters least.\nFirst one set for everybody, then eight Multi-query attention takes the observation at full strength. Every head keeps its own query projection; the whole layer gets one key projection and one value projection, shared. Each head still asks its own question and produces its own output. The cache falls by exactly the head count — thirty-two heads, one thirty-second of the storage, a sixteen-gigabyte cache reduced to half a gigabyte.\nThen the honest cost, measurable rather than theoretical. Quality dropped, modestly but detectably, and models trained this way from scratch were reported to be less stable during training. Neither problem was bad enough to abandon the idea, and both were bad enough that nobody wanted to ship it.\nGrouped-query attention is the compromise that did ship. Divide the query heads into groups and give each group its own key and value projection: sixty-four heads as eight groups of eight. The cache falls by the ratio of groups to heads, one-eighth here, for a quality loss that careful evaluation finds and ordinary use does not. Essentially every model released since has been built this way.\nIt is not a new mechanism but an interpolation between two endpoints already in hand. The interesting part is the shape of the curve, which is remarkably flat. Sixty-four groups down to eight discards seven-eighths of the cache and costs little; eight down to one discards a further eighth and costs more than the seven-eighths did. The last saving is the expensive one, and nobody has a satisfying account of where the knee sits. The redundancy is real but not total, and eight is roughly where it runs out.\nUptraining is why this spread in months rather than years. A finished multi-head model can be converted rather than replaced: average the key projections of each group into one matrix, the values likewise, and the grouped architecture arrives with a serviceable initialization. Averaging blurs whatever the individual heads were doing differently, so the converted model is worse than its parent — and a short continuation of training, a few percent of the original budget, recovers most of the loss. Thirty-two near-identical photocopies become eight, and nobody had to gamble a from-scratch run on whether the trade was worth taking.\nFigure 1. MHA, GQA and MQA, and what each caches.\nFigure 2. The saving is nearly free until it is not.\nA pinch of math: the ratio of groups to heads The cache is what is being attacked, so write down its size. Per token, each layer stores one key and one value for every key-value head, each as wide as a head:\n$$\\text{numbers per token} = 2 \\cdot L \\cdot H \\cdot d_h$$The $2$ is for keys and values, $L$ is the layer count, $H$ the number of heads, and $d_h$ the width of one head. Sharing replaces $H$ with the number of groups $G$:\n$$\\text{numbers per token} = 2 \\cdot L \\cdot G \\cdot d_h$$so the ratio of the two is $G/H$ and nothing else; no other factor has moved. Put a large model in: eighty layers, sixty-four heads of width $128$, two bytes per number, about two and a half megabytes per token. At a context of a hundred and thirty thousand tokens, one conversation carries roughly three hundred and twenty gigabytes, more than any single accelerator holds. With eight groups it carries forty. With one group, five. Stack cache quantization on top and the factors multiply.\nNow count the arithmetic. The query count has not changed: $H$ query heads still dot against every stored position, so the score grid is exactly $H$ by $n$ as before and the multiply-adds are identical. Each key-value pair is simply read by $H/G$ query heads instead of one. The same arithmetic, on one-eighth of the bytes — and since decode waits on memory rather than on the arithmetic units, that ratio decides how long it waits. One factor in a product moved and the flop count did not notice.\nFigure 3. One conversation\u0026rsquo;s cache at 130,000 tokens.\nWhat it does not fix Three limits and one rival, stated plainly, because the technique is routinely credited with more than it does.\nIt saves memory, not computation. The grid of scores is the size it always was, so the quadratic term is untouched and prefill costs what it cost before: eight groups read a hundred-thousand-token prompt at the price of sixty-four. It does nothing for the compute side of long context , nothing that overlaps with the tiling in FlashAttention , and nothing for training. These compose; none substitutes for another.\nIt reduces expressiveness by construction, and the loss is small rather than zero. Eight key spaces exist where sixty-four did. Heads in a group can still ask different questions, but only of one shared set of keys, and a relationship needing a key space no sibling head wanted has nowhere to live. Losses of this kind average away in aggregate benchmarks and surface in the particular capability someone is relying on.\nIt interacts awkwardly with tensor parallelism . Splitting an attention layer across devices normally gives each device a subset of the heads. A group count below the device count means the key and value projections cannot be divided that way and must be replicated instead, along with the cache they produce, handing back part of the saving where memory was tightest. Eight groups across eight devices is clean; eight across sixteen is not, which is why group counts are chosen with serving topologies in mind.\nA later line of work attacks the same constraint without sharing anything. Rather than forcing heads onto identical keys and values, it projects keys and values jointly into a small shared latent, caches the latent, and reconstructs what each head needs at attention time. Nothing must be equal, only to lie in a low-dimensional subspace, which recovers more quality per stored byte than sharing does. The cost is a considerably more complicated kernel. Both are in production at scale, and which is preferable is unsettled.\nWhat to carry forward Attention\u0026rsquo;s cost grows with the square of the length, but that square is a training and prefill cost; during generation the row of scores is one token wide and what fills the machine is the cache of one key and one value per token, per head, per layer, per conversation. Weights are read once per batch and shared by all of it; a cache is read by one sequence, so the cache rather than the parameter count sets how many users a machine holds, which sets throughput, which sets the price per token. The technique exploits an asymmetry inside the head: each head has its own query, key and value projection, the query carries the specialization, and keys and values are largely redundant across heads. So: a query per head, shared keys and values. Multi-query attention shares one key-value pair across all heads and divides the cache by the head count, at a cost of measurable quality loss and instability. Grouped-query attention interpolates — heads divided into groups, one key-value pair per group, commonly eight groups for sixty-four heads — keeping most of the memory saving and most of the quality; the notable fact is how flat that curve is, the last portion of the saving costing more than all the rest. An existing multi-head model can be converted by averaging the key and value projections within each group and training on a fraction of the original budget, hence the speed of adoption. It does not touch arithmetic: score grid and prefill cost are unchanged, expressiveness is reduced by a small but non-zero amount, and a group count below the device count forces the shared projections to be replicated rather than split. What it buys is concurrency in the one resource that was running out, and nothing else.\nThe figures in this article follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/articles/grouped-query-attention/","summary":"\u003cp\u003eThe usual account of attention\u0026rsquo;s cost is that it grows with the square of the sequence length, and the well-known responses are aimed at that square. A machine serving a large model to a few hundred people at once is not usually failing on arithmetic. It is failing on space. Somewhere in its memory sits one key vector and one value vector for every token, of every head, in every layer, for every conversation in flight, and that structure — not the parameter count, not the score grid — decides how many people the machine can hold and therefore what a token costs to sell. The remedy has to be storing less per token. The technique that delivered it rests on an observation close to an insult: most of what is stored is redundant. This piece is about which part of an attention head carries its specialization, what it costs to make heads share the rest, and why the saving turns out to be so nearly free.\u003c/p\u003e","title":"Grouped-Query Attention — Paying for Fewer Copies of the Same Notes"},{"content":"Ask a model for the page number of a quotation it has never seen and it will give you one. Not a hedge, not a refusal, not a note that the request is beyond it — a page number, formatted correctly, inside a sentence of ordinary competent prose. The common reading is that the model is guessing badly, or was trained carelessly, or is in some loose sense pretending. None of those is quite right. The behaviour is what the training procedure asks for, at every position, and a model that did otherwise would have scored worse for it. That turns hallucination from a mysterious defect into a consequence of what the objective measures and what the available text contains. This piece is about why abstention is never the rewarded action, what the model\u0026rsquo;s own probabilities do and do not tell you about its uncertainty, and which of the available interventions actually change the situation.\nThe objective never rewards a refusal Start with what the loss measures, because most accounts describe it loosely and the looseness hides the mechanism. During pretraining the model produces, at every position in a stretch of text, a distribution over the next token; the cross-entropy loss there is the negative logarithm of the probability it assigned to the token that actually came next. Every position is graded, on exactly one thing: the mass placed on the observed continuation.\nNow go looking for the abstention. The vocabulary holds the pieces of \u0026ldquo;I do not know\u0026rdquo;, so the model can certainly emit that string. What it cannot do is be rewarded for emitting it where the text went on to say something else, because there the target is whatever the writer wrote, and mass spent on a hedge is mass taken from the target. Abstention is not merely unrewarded. It is penalized, whenever the text in front of it went on to make a claim.\nThe consequence is sharper than it first looks. Take a model completing a sentence that requires a citation it does not hold. If it emits a plausible-looking volume and page, the loss is the loss for getting those tokens wrong — precisely what a differently invented, equally plausible pair would have received. Nothing separates the two, and nothing charges either for the act of inventing. Fluent continuation is the whole of what is optimized. Accuracy is something it correlates with, on material the model absorbed, and correlation is all it ever is.\nNobody writes down what they declined to say There is a second version of the argument, stronger than the first, and it concerns the data rather than the loss.\nAt training time the model is graded against text written by someone who knew the answer. That sounds obvious; its implications are not. A reference article states a population figure because its author had it to hand; a forum post about a compiler error was written by someone who had already found the cause. Where a writer did not know, the usual outcome is not a published sentence saying so but a sentence never written, a paragraph cut, a draft left in a drawer. The corpus is what survived a filter that removed most of the ignorance before publication.\nSo the behaviour a user wants — confidently declining a question the model nearly knows — has almost no support in the data. The model did not see that behaviour and learn to avoid it. There was very little of it there to see.\nHedging exists in text, of course, and this is where the point turns uncomfortable rather than merely surprising. Academic prose hedges constantly, forum answers say I think. Those hedges are properties of genre, not reports of a writer\u0026rsquo;s uncertainty about the fact at hand. A model that learns them acquires a register it can adopt on stylistic cues: when hedging sounds appropriate, which is a different function from whether this claim can be supported.\nFigure 1. Why no path through training rewards a hedge.\nCalibration is a real signal about the wrong object The model is not wholly without information about its own uncertainty. The kind it has matters.\nCalibration asks whether stated probabilities match observed frequencies: across the cases where a predictor said seventy percent, about seventy percent should have happened. Base models — the raw products of pretraining, before instruction tuning — are often reasonably calibrated over next tokens, which is unsurprising given a loss that rewards honest reporting. The information about uncertainty exists somewhere in the output distribution rather than nowhere at all.\nThen the costs. A single token\u0026rsquo;s probability is not the probability that a claim is true. The quantity a user needs is over a proposition — that this person held that office in that year — and what the model produces is over tokens given a prefix. Different objects. A low probability may reflect only a choice among synonyms, with no doubt about the fact. A high probability may sit on a token that is merely the obligatory continuation of a sentence already committed to a falsehood. Getting from tokens to propositions means summing over every sequence expressing the claim, and generation does no summing.\nPost-training tends to make the signal worse. Preference data comes from people comparing answers, and people reliably prefer answers that sound confident. A preference-optimized model is pushed toward the surface features hedging lacks, and the optimization obliges: instruction-tuned models are routinely more overconfident than the base models beneath them. The stage did what it was asked and degraded a property nobody measured.\nMore capable models relocate the problem rather than removing it. A stronger model hallucinates less on material it absorbed well, and more convincingly on material it did not: the invented citation acquires a real journal and an author who works in the field. So the error rate improves while the detectability of the errors that remain gets worse, moving in opposite directions on a quantity a reader depends on. An uncomfortable trade, and no phrasing of it reads well.\nThree failures reported as one number The word hallucination covers at least three mechanisms that share a symptom and little else.\nThere are failures of retrieval, where the material is in the weights and the wrong thing came out — the right sort of answer attached to the wrong entity, two biographies merged, a date shifted by a year. Memorization is partial and lossy, and these resemble the failures of a person recalling something read once.\nThere are failures of absence, where the material was never in the weights. Nothing was recalled because there was nothing to recall, and what appears instead is the shape such an answer takes, filled with tokens that fit it.\nAnd there are failures of grounding, where a source was supplied and the output does not follow from it. The document said one thing, the answer says another, and the answer may even be true while unsupported by the material it claims to summarize.\nThese have different remedies, which is why separating them matters, and they are routinely reported as one hallucination rate. A number that improves may reflect better recall, broader coverage, or more faithful reading, and the evaluation rarely says which.\nRetrieval changes the category rather than closing it. Putting relevant documents in the prompt, as retrieval-augmented generation does, converts absence into grounding. That is progress, since grounding failures can be checked against a source and absence failures cannot. It is not a solution, and the successor failure is specific: confident misreading of a supplied document, harder to notice than a fabrication because the citation resolves.\nSampling several answers detects instability, not error. Generate an answer several times at nonzero temperature, which the decoding settings make trivial, and see whether the versions agree. Disagreement is informative; a model offering four different dates is saying something useful. Agreement is not the complement of it. A model wrong the same way every time, which is what a systematic gap in the data produces, looks exactly like a model that is right.\nTraining an abstention installs the decline and its cost together. A model can be taught to say it does not know, through preference methods on data where declining is favoured, or a verifier rewarding refusal on unanswerable items. This works. What it buys is a decline that also fires on things the model did know, because the trained boundary is not the true boundary of its knowledge, and no placement avoids both errors at once. It is the two-sided trade any refusal carries.\nFigure 3. Three failures, three remedies, one reported number.\nA pinch of math: why abstention is never the optimal move Two small calculations, pointing the same way.\nThe first acquits the objective. Write $q$ for the true conditional distribution over the next token in some context, and $p$ for the model\u0026rsquo;s. The expected loss incurred at that position is\n$$\\mathbb{E}_q[-\\log p] = -\\sum_{i} q_i \\log p_i$$and this is a proper scoring rule: over all choices of $p$, it is minimized exactly when $p = q$. A model that shades its probabilities toward what sounds good does worse than one reporting what it estimates. The loss does not ask for overconfidence.\nThe trouble lies in what $q$ is defined over. That sum runs across the vocabulary, which contains no element meaning this question has no answer available to me. Whatever $q$ is, it is a distribution over continuations of written text, and in a corpus produced by people who knew things, the continuation at a position requiring a fact is the fact. There is nothing to put mass on that expresses ignorance.\nThe second calculation says where the damage lands. Suppose a claim occupies $k$ tokens, each drawn correctly with probability $p$, independently. The chance the whole claim is right is\n$$p^k$$With $p = 0.95$ and $k = 20$, that is about $0.36$. The independence assumption is generous — tokens are correlated, and a model can recover from a bad one — but the exponent is the point. A one-word answer survives a moderate per-token error rate; a full citation or a case number does not. This is why fabrication clusters on names, numbers, dates and references rather than on the surrounding argument — on exactly the details a reader is most likely to check.\nFigure 2. Why fabrication clusters on names and numbers.\nWhat to carry forward The training loss grades a model at every position on the probability it gave the token that actually came next, and no token in the vocabulary meaning I do not know would ever be the correct target in ordinary text, so abstention is penalized wherever the text went on to make a claim; a fabricated citation is charged exactly what a differently fabricated one would be, and neither is charged for the fabricating. The sharper form concerns the data: the corpus was written by people who knew the answers, since sentences composed out of ignorance rarely got published, so confidently declining a question one nearly knows is a behaviour with almost nothing to imitate — and the hedging that does appear belongs to genre rather than reporting on whether this claim can be supported. Uncertainty information does exist in the output distribution, and base models are often reasonably calibrated over next tokens, but a token\u0026rsquo;s probability is not the probability a proposition is true, and preference optimization erodes the calibration there was, since raters favour confident-sounding answers. Capability dissolves none of it: stronger models err less on well-covered material and err more convincingly elsewhere, so rate and detectability move in opposite directions. Failures of retrieval, of absence, and of grounding carry distinct remedies and one shared reported number. Retrieval converts absence into grounding, resampling detects instability but not systematic error, and a trained refusal declines some things the model did know. This is a property of the objective and of the available text rather than a defect of any particular model, so it is mitigated and not removed, and every mitigation is paid for in helpfulness.\nThe figures in this article follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/articles/hallucination/","summary":"\u003cp\u003eAsk a model for the page number of a quotation it has never seen and it will give you one. Not a hedge, not a refusal, not a note that the request is beyond it — a page number, formatted correctly, inside a sentence of ordinary competent prose. The common reading is that the model is guessing badly, or was trained carelessly, or is in some loose sense pretending. None of those is quite right. The behaviour is what the training procedure asks for, at every position, and a model that did otherwise would have scored worse for it. That turns \u003cem\u003ehallucination\u003c/em\u003e from a mysterious defect into a consequence of what the objective measures and what the available text contains. This piece is about why abstention is never the rewarded action, what the model\u0026rsquo;s own probabilities do and do not tell you about its uncertainty, and which of the available interventions actually change the situation.\u003c/p\u003e","title":"Hallucination — Why a Model Cannot Tell You What It Does Not Know"},{"content":"Most claims about what happens inside a language model are inferences from behaviour: the model does this, so something in it must be doing that. One case is different. There is a mechanism whose components have been located, whose wiring has been traced, whose function has been stated precisely, and which can be switched off to confirm that the behaviour it explains disappears with it. It is not complicated — two attention heads in sequence, performing a copy — and it accounts for a surprising share of what makes these models useful, including the ability to follow a pattern demonstrated in a prompt. It also arrives during training in a way that is visible from outside, in a small kink in the loss curve that turns out to mark the moment a capability comes into being. This piece is about that mechanism and what its existence suggests.\nThe behaviour to be explained Give a model a sequence in which something has already happened once, and it will predict that it happens the same way again. Show it ... Dr. Aisha Rahman ... Dr. Aisha and it will produce Rahman. Show it a list where every entry follows a template and it will continue the template. Give it three examples of a made-up transformation and it will apply the transformation to a fourth.\nThe general rule is: find where this current token occurred earlier in the context, and predict whatever followed it then. Stated so plainly it sounds trivial, and it is not, for two reasons. The name being copied may be one the model has never seen — invented for the occasion — so this cannot be retrieval from training. And it is the mechanical core of in-context learning: a model that can find a previous instance of the current situation and repeat what came next is a model that can be taught by demonstration inside a prompt, without any weights changing.\nWhy it takes two heads A single attention head cannot do this, and understanding why is the crux.\nSuppose the current token is Aisha and we want to find the earlier Aisha and copy what came after it. A head attending directly to the earlier Aisha retrieves that token\u0026rsquo;s value — which tells us about Aisha, which we already know. We do not want the matching token. We want its successor.\nThe head could try attending to Rahman directly, but on what basis? Rahman is not the token we are holding and there is nothing about it to match on. The information required — that Rahman is preceded by Aisha — is not present at Rahman\u0026rsquo;s position. It has to be put there.\nSo the mechanism is built in two stages, in different layers.\nA previous-token head, in an early layer, does something almost aggressively simple: at every position, attend to the position immediately before and copy information about it into the residual stream . After this layer, the position holding Rahman also carries a record that its predecessor was Aisha. Every position now knows what came before it.\nAn induction head, in a later layer, does the actual work. Its query, formed from the current token Aisha, is matched not against other tokens\u0026rsquo; identities but against the previous-token record that the earlier head deposited. So it asks: which position is preceded by Aisha? That question matches at Rahman\u0026rsquo;s position. The head attends there, copies Rahman\u0026rsquo;s value forward, and the unembedding turns it into a prediction of Rahman.\nThe two heads form a circuit: distinct components in different layers, communicating through the residual stream, jointly implementing a function that neither performs alone. The first head writes into a subspace; the second reads that same subspace. This is precisely the mechanism described in the article on the residual stream — components coordinating across depth by agreeing on a direction — and it is the clearest worked example of it.\nFigure 1. The two-head circuit that copies a successor.\nA pinch of math: matching on a shifted key The composition can be made precise, and it is worth doing because it shows the trick is one substitution rather than anything elaborate.\nAn ordinary head computes attention scores between position $i$ and position $j$ as\n$$s_{ij} = (W_Q x_i) \\cdot (W_K x_j)$$where $x$ is the residual stream at that position. If the stream mostly carries token identity, this matches token against token — useful for many things, useless here.\nThe previous-token head has modified the stream. At position $j$ it has added a term encoding the identity of token $j-1$, along some direction the head writes into. Write the stream as approximately $x_j \\approx e_j + \\tilde{e}_{j-1}$, where $e$ denotes the token\u0026rsquo;s own contribution and $\\tilde{e}$ the shifted copy in its own subspace.\nNow the induction head\u0026rsquo;s key matrix $W_K$ is trained to read the shifted subspace and ignore the direct one, while its query matrix $W_Q$ reads the direct one. The score becomes\n$$s_{ij} \\;\\approx\\; (W_Q e_i) \\cdot (W_K \\tilde{e}_{j-1})$$which is large when $e_i$ and $e_{j-1}$ describe the same token. The head attends to position $j$ when position $j-1$ matched the current token — exactly the required behaviour.\nThe remaining piece is the copying. The head\u0026rsquo;s value and output matrices are arranged so that whatever token is at the attended position is written back into the stream in a direction that promotes that same token in the output. Composing the two matrices gives an operator on token space whose eigenvalues skew positive: read a token, promote that token. This can be inspected directly in a trained model, and the fraction of the eigenvalue mass that is positive serves as a quantitative signature distinguishing real copying heads from heads that merely look similar. Note the condition is only that the spectrum leans positive — not that the operator approximates the identity, which would be far stronger. In large models with feed-forward layers this weight-based analysis becomes unwieldy, and heads are identified behaviourally instead, by a copying score and a prefix-matching score measured on activations.\nThe bump in the loss curve The most striking thing about these heads is that you can watch them form.\nEarly in training, the loss falls smoothly. Then, over a narrow window of a few hundred steps, there is a small bump — a brief plateau or even a slight rise, followed by a faster drop. It is easy to miss and appears reliably across model sizes, and it had no accepted explanation for some time.\nIt is the induction heads forming. Before the window, the model has no way to use repeated structure in its context. During it, the two-stage circuit assembles. After it, the model can copy from context, and its loss drops because a great deal of text contains repetition that was previously unexploitable.\nTwo things make this more than a curiosity. First, in-context learning ability — measured as how much better the model predicts late tokens in a sequence than early ones, which is precisely the benefit of having context to learn from — appears at the same moment. The capability and the mechanism arrive together. Second, ablating the induction heads removes most of that ability — though this is where the account is weaker than it is usually reported to be. The ablations were run on small models, and are strongest for attention-only ones; no ablation was performed at production scale, and for large models the evidence is correlational and mechanistic rather than causal. The authors themselves note the co-occurrence could reflect a shared cause, such as the model learning to compose layers through the residual stream at that moment. Taken at small scale it is still about as complete a story as interpretability has produced about anything: a behaviour, a mechanism, a moment of formation, and an intervention that removes both together.\nIt also gives a concrete answer to a question that is otherwise vague. Capabilities that appear to emerge suddenly with scale or training may correspond to specific circuits assembling at specific moments. Not all of them will — and the article on emergent abilities sets out the good deflationary argument that much apparent emergence is a measurement artifact — but here is one case where a discontinuity in behaviour really does correspond to a discrete thing coming into existence.\nFigure 2. A bump in the loss, and a capability arriving.\nWhat generalizes, and what does not The simple copy is the beginning rather than the end. In larger models the same machinery is found doing more abstract versions of the same operation: matching not on exact token identity but on similarity, so the pattern completes for a paraphrase rather than a repeat; matching on position within a repeated structure, so a template is followed rather than a string; operating over several tokens rather than one. The family is broad and the boundaries are not crisp.\nRelated circuits have been traced by the same methods. The most thoroughly documented is the one behind indirect object identification — deciding, in when Mary and John went to the shop, John handed a drink to ___, that the answer is Mary — which involves a chain of head types including some that identify names, some that suppress duplicates, and the name-mover heads that do the final copying. That is the circuit reconstructed in the worked example of the article on ablation , and it was found with the same combination of ablation and patching.\nThe honest limit is scale. These circuits were mapped in small models, on narrow tasks, by people spending months on each. Nobody has a full account of a production model, and the reason is superposition : components in a large model are polysemantic, participating in many circuits at once, so isolating one is far harder than in a two-layer model built for study. The programme of finding circuits by hand does not obviously scale, which is why the field has shifted toward automated decomposition — sparse autoencoders to find the features first, and automated search for the connections between them. Induction heads remain the proof that there is something there to find.\nWhat to carry forward An induction head implements a rule of the form: find where the current token appeared earlier in the context, and predict whatever followed it then. This works on tokens the model has never seen, so it is not retrieval from training, and it is the mechanical core of learning from examples inside a prompt. It requires two heads because a head attending to the earlier occurrence retrieves that token rather than its successor, and the fact that a given position is preceded by a particular token is not available at that position until something puts it there — so a previous-token head in an early layer copies each position\u0026rsquo;s predecessor into the residual stream , and an induction head in a later layer forms its query from the current token and matches it against that shifted record, attending to whatever position was preceded by a match and copying it forward. The two communicate through a shared subspace, which is the clearest worked example of a circuit. The composed value-and-output operator of a real induction head has eigenvalues that skew positive — read a token, promote that token — which serves as a quantitative signature in small models, though at scale heads are identified behaviourally instead. The heads form abruptly during training, producing a visible bump in the loss curve, and in-context learning ability appears in the same window and disappears when they are ablated in small models — a causal account at that scale, and a correlational one at production scale, where the ablations were never run. Larger models contain more abstract versions matching on similarity or structure rather than identity, and related circuits have been mapped by the same methods — but all of it has been done in small models on narrow tasks, and the polysemanticity of large ones is why the field has turned toward automated decomposition instead.\nThe figures in this article follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/articles/induction-heads/","summary":"\u003cp\u003eMost claims about what happens inside a language model are inferences from behaviour: the model does this, so something in it must be doing that. One case is different. There is a mechanism whose components have been located, whose wiring has been traced, whose function has been stated precisely, and which can be switched off to confirm that the behaviour it explains disappears with it. It is not complicated — two attention heads in sequence, performing a copy — and it accounts for a surprising share of what makes these models useful, including the ability to follow a pattern demonstrated in a prompt. It also arrives during training in a way that is visible from outside, in a small kink in the loss curve that turns out to mark the moment a capability comes into being. This piece is about that mechanism and what its existence suggests.\u003c/p\u003e","title":"Induction Heads — The First Circuit Anyone Fully Understood"},{"content":"QLoRA compressed a frozen model so that adapters could be trained beside it, and mixed precision worked out how few bits a model can be trained in. The third case is the one most people actually encounter: taking a model that is finished and will never be trained again, and making it smaller so that it fits on hardware someone owns. The situation is more forgiving than training, since there are no gradients to preserve and no accumulation to worry about — but it is also less forgiving in one specific way, because there is no opportunity to adapt. Whatever damage the compression does is permanent, and the model must live with it on every request. This piece is about how much can be taken away, why the activations turn out to be harder than the weights, and what the resulting formats actually buy.\nWhy bother, precisely The motivation is not storage. It is the bandwidth argument that has now appeared several times: generating a token requires streaming every weight out of memory, and the rate at which that can be done sets the speed. Halve the size of the weights and you halve the traffic, and since the arithmetic units were idle anyway, you approximately double the generation speed. Quantization is a throughput technique that happens to also reduce memory.\nThe memory reduction is what determines feasibility, though. A model in half precision needs about two bytes per parameter, so a seventy-billion-parameter model wants a hundred and forty gigabytes and does not fit on any single accelerator. At four bits it wants thirty-five, which fits on one. The difference is not incremental; it is the difference between needing a multi-device server and needing a workstation, and it is why quantization is the single technique most responsible for open-weight models being usable by individuals.\nFigure 1. What seventy billion parameters cost per bit-width.\nRounding a weight matrix The basic operation is the same as in QLoRA. Take a group of numbers, find their range, divide that range into a small number of levels, and record which level each number is nearest to along with the scale needed to recover it.\n$$w \\;\\approx\\; s \\cdot \\text{round}\\!\\left(\\frac{w}{s}\\right), \\qquad s = \\frac{\\max|w|}{2^{b-1} - 1}$$Everything of consequence is in how the groups are chosen. One scale for an entire matrix is cheapest and worst, because a single outlier stretches the range and crushes everything else into a few levels. A scale per row is better. A scale per block of thirty-two or sixty-four consecutive values is better again, and is what current formats use — the scales themselves become a meaningful overhead at that granularity, so they are often stored in reduced precision too, which is the double-quantization idea from QLoRA.\nAt eight bits this is nearly free: round-to-nearest with per-row scales loses essentially nothing measurable, and it can be done in seconds with no data and no reasoning required. At four bits, naive rounding starts to hurt, and the interesting methods begin.\nThe insight: not all weights matter equally The move that separates a good four-bit model from a mediocre one is to stop treating quantization as a problem about weights and start treating it as a problem about outputs.\nThe goal was never to represent the weights accurately. It was to make the layer produce the same answers. A weight that is multiplied by a large activation on typical inputs affects the output a great deal, and an error in it is expensive; a weight that is multiplied by something near zero can be rounded carelessly at no cost. So the error to minimize is not $\\|W - \\hat{W}\\|$ but $\\|Wx - \\hat{W}x\\|$ over realistic $x$ — which requires knowing what realistic $x$ looks like, and therefore requires a calibration set: a small sample of representative text, typically a few hundred sequences, pushed through the model to observe the activations each layer actually sees.\nTwo families of method exploit this, and they differ in how.\nOne works sequentially within each matrix. Quantize the weights one column at a time; after each, compute the error introduced and adjust the remaining un-quantized weights to compensate for it. Errors are not merely accepted but partially cancelled by the columns that follow, using second-order information about how the layer\u0026rsquo;s output responds to each weight. It is a careful, slightly expensive procedure — a few GPU-hours for a large model — and it produces good four-bit results.\nThe other works by rescaling before rounding. The observation is that the damage is dominated by a small fraction of weight channels, identifiable by the magnitude of the activations flowing through them rather than by the magnitude of the weights themselves. Multiply those channels up before quantizing, so they land on a finer part of the grid, and divide the corresponding activations down to compensate — the product is unchanged, and the important channels are represented more precisely. It is simpler, faster, and does not modify weights based on a specific calibration text, which makes it less prone to overfitting the calibration set.\nBoth reach roughly the same place: four-bit weights with quality close enough to the original that ordinary use does not reveal a difference, though careful evaluation still can.\nWhy the activations are harder Everything above quantizes weights while computing in higher precision — the weights are stored small and expanded on the fly, which is the store-narrow-compute-wide pattern running through all of this. That halves the memory traffic but does not use faster arithmetic, since the multiplication still happens in sixteen bits.\nTo get the arithmetic speedup as well, the activations must be quantized too, and here a genuine obstacle appears. Weights are well-behaved — roughly bell-shaped, similar in scale across a matrix. Activations are not. A small number of dimensions in the residual stream carry values orders of magnitude larger than the rest, and they appear in the same dimensions for every token.\nThese are the outliers described in the article on softmax , and their origin is now understood: they are largely the machinery supporting attention sinks, the mechanism by which a head that has nothing to do parks its obligatory unit of attention somewhere harmless. Making that work requires a few directions with very large magnitudes, and those directions ruin any quantization scheme that assigns a shared scale to a whole tensor — the scale must accommodate the outlier, and everything else collapses into a handful of levels.\nThe workarounds all amount to keeping the outliers out of the general scheme. Identify the offending dimensions and compute those in full precision while quantizing the rest. Or move the difficulty: since the layer computes $Wx$, the scale can be shifted from the activations into the weights by dividing $x$ and multiplying $W$ correspondingly, making the activations easy to quantize at the cost of making the weights slightly harder — a good trade, because weights were the easy part. Or rotate the representation before quantizing, using a transformation that spreads the outlier\u0026rsquo;s magnitude across many dimensions so that no single one is extreme; this can be folded into the surrounding matrices so it costs nothing at runtime, and it is the cleanest of the three because it removes the outliers rather than exempting them.\nFigure 2. Weights are bell-shaped; activations are not.\nWhere the floor is Four bits is comfortable. Three is noticeably degraded. Two, by direct rounding, destroys the model.\nThe lower range is nonetheless active, and the reason is a result from the scaling laws direction: at a fixed memory budget, a larger model quantized more aggressively often beats a smaller model quantized gently. If you have thirty-five gigabytes, a seventy-billion model at four bits generally outperforms a thirteen-billion model at sixteen. Parameters are worth more than precision, within limits, which pushes practitioners toward the largest model that will fit at whatever bit-width still works.\nPushing below three requires giving up round-to-nearest entirely. The successful approaches replace each group of weights with an index into a learned codebook of vectors, so the representation is a lookup rather than a grid — vector quantization, which exploits the fact that groups of weights are correlated and that a well-chosen set of a few hundred representative vectors covers the space far better than uniform levels do. This reaches usable quality near two bits, at the cost of a more complex and slower decode path.\nAnd there is one thing quantization cannot fix. The KV cache is not weights, and shrinking the weights does nothing to it — so a heavily quantized model serving long conversations finds that the cache, untouched, now dominates its memory entirely. Quantizing the cache is a separate exercise with its own difficulties, keys being noticeably less tolerant than values.\nFigure 3. More parameters beats more precision, up to a floor.\nWhat to carry forward Quantizing a finished model is easier than quantizing one being trained, since no gradients must survive and nothing accumulates — but the damage is permanent, because there is no chance to adapt around it. The motivation is bandwidth as much as memory: generation streams every weight from memory to produce each token, so halving the weights roughly doubles the speed, while the memory saving is what decides whether the model fits on hardware at all. The mechanics are rounding to a grid with a scale per small block of values, which is nearly free at eight bits and needs help at four. That help comes from reframing the goal: the aim is not to represent the weights accurately but to preserve the layer\u0026rsquo;s outputs, which requires a calibration sample to learn what activations each layer actually sees. Two families follow — one quantizing column by column and adjusting the remaining weights to cancel the error already introduced, the other rescaling the channels that carry large activations so they land on a finer part of the grid. Quantizing the activations as well, which is what unlocks faster arithmetic rather than merely less traffic, runs into the outlier dimensions that exist to support attention sinks; the fixes exempt them, shift their scale into the weights, or rotate the representation so no single dimension is extreme. Four bits is comfortable, three is degraded, and below that requires codebooks rather than grids — and throughout, a larger model quantized harder tends to beat a smaller one quantized gently at equal memory. None of it touches the cache, which is untouched by weight compression and becomes the dominant cost once the weights have shrunk.\nThe figures in this article follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/articles/inference-quantization/","summary":"\u003cp\u003e\u003ca href=\"/articles/qlora/\"\u003eQLoRA\u003c/a\u003e\n compressed a frozen model so that adapters could be trained beside it, and \u003ca href=\"/articles/mixed-precision/\"\u003emixed precision\u003c/a\u003e\n worked out how few bits a model can be trained in. The third case is the one most people actually encounter: taking a model that is finished and will never be trained again, and making it smaller so that it fits on hardware someone owns. The situation is more forgiving than training, since there are no gradients to preserve and no accumulation to worry about — but it is also less forgiving in one specific way, because there is no opportunity to adapt. Whatever damage the compression does is permanent, and the model must live with it on every request. This piece is about how much can be taken away, why the \u003cem\u003eactivations\u003c/em\u003e turn out to be harder than the weights, and what the resulting formats actually buy.\u003c/p\u003e","title":"Inference Quantization — Shrinking a Finished Model to Fit"},{"content":"Before the first gradient arrives, every weight in the model has to be set to something, and the something is drawn at random. This sounds like a detail to be got out of the way, and for a long time it was treated as one. It is not. The scale of those random draws determines whether a signal survives its journey up an eighty-block tower or dies on the way; it determines whether the gradient makes it back down; and — the part that turns a numerical nicety into an economic question — it determines whether a learning rate that worked on a small model still works on a large one. That last property has a name and a construction, and it is the reason nobody tunes hyperparameters on a frontier model directly. This piece is about how the starting point is chosen and why choosing it correctly lets you do your experiments somewhere cheap.\nThe problem with getting the scale wrong Take a vector, multiply it by a random matrix, multiply the result by another, and keep going. What happens to its magnitude depends entirely on how large the random entries are.\nIf they are slightly too large, each multiplication amplifies, and eighty amplifications compound into a number the arithmetic cannot represent. If they are slightly too small, each multiplication shrinks, and eighty shrinkages leave a signal indistinguishable from zero. The tolerance is narrow, and it narrows further with depth, because the effect is multiplicative. There is no learning to be done in either regime: the forward pass produces garbage or nothing, and the backward pass, which is subject to the same compounding, delivers a gradient that has exploded or vanished before it reaches the lower layers.\nThe classical fix reasons about variance. If a layer takes $n_{\\text{in}}$ inputs and sums them with random weights, and we want the output to have roughly the same spread as the input, then the variance of each weight should scale as $1/n_{\\text{in}}$ — because summing $n$ independent terms multiplies the variance by $n$, and we need to cancel that. Drawing weights from a distribution with\n$$\\text{Var}(W_{ij}) = \\frac{1}{n_{\\text{in}}}$$keeps activations at a stable scale layer after layer. Variants differ in whether they also account for the backward pass, which suggests $1/n_{\\text{out}}$, and in how they handle the shrinkage caused by a nonlinearity that zeroes half its inputs, which suggests a factor of two. These are the schemes named after their authors, and for a plain deep network they are close to the end of the story.\nFor a transformer there are two amendments. Because every block adds into the residual stream and nothing subtracts, the accumulated magnitude grows with depth, so each block\u0026rsquo;s output projection is usually scaled down by a factor involving the number of layers — commonly dividing by the square root of the depth — so that the total after all blocks stays sane. And a few specific matrices are deliberately started at zero rather than at random: the second matrix of each LoRA pair, so that adaptation begins as the identity, and in some designs a learned per-block gain, so that the model begins as a stack of blocks that do nothing and switches them on as it finds a use for them.\nThe problem this does not solve Standard initialization keeps a given model trainable. It does not make two models of different sizes behave alike, and that is a separate and more expensive problem.\nHere is the situation anyone training a large model faces. The learning rate is the most consequential dial there is, and it has no theory that fixes it — it has to be found empirically. Finding it means training a model several times at different values and seeing which works. On a model that costs several million dollars and two months per run, that is not an experiment anybody performs. So the learning rate is tuned on a small model and reused on the large one.\nUnder standard initialization this transfer does not work. The optimal learning rate shifts systematically as width grows, and it shifts downward: a value that trains a small model beautifully will destabilize a wide one. Practitioners knew this and compensated with rules of thumb and with warm-up, but the compensation was folklore, and it meant every large run began with an educated guess that could not be validated until it was too late to change. A meaningful fraction of the compute spent on large models in that era went on runs that diverged.\nWhy does the optimum move? Because standard initialization keeps the forward activations stable at initialization and says nothing about what an update does. Consider one layer\u0026rsquo;s contribution: as width $n$ grows, an update to that layer changes its output by an amount that also scales with $n$, since more terms are being summed. A step size appropriate at width one thousand therefore produces a change several times too large at width eight thousand. The forward pass was made width-independent; the update was not.\nMaximal update parameterization The fix is to work out, for every matrix in the model, how its initialization scale and its learning rate must depend on width so that three things hold simultaneously as width grows: activations stay a constant size, the change in activations caused by one update stays a constant size, and the logits stay a constant size. Impose all three and the scalings are determined — there is essentially one answer, and it is called maximal update parameterization, usually shortened to muP.\nThe construction sorts the model\u0026rsquo;s matrices into three classes, and the classification is the substance of it.\nMatrices whose input width is fixed and whose output width grows — the embedding table, whose input is a one-hot over a fixed vocabulary — keep an initialization variance that does not shrink with width, and a learning rate that does not shrink either.\nMatrices where both dimensions grow with width — everything in the interior of the model — get the usual $1/n$ initialization variance, but their learning rate is scaled down proportionally to $1/n$. This is the crucial departure, and it is what standard practice was missing.\nThe output matrix, whose input width grows and whose output is a fixed vocabulary, gets its own treatment: its contribution is scaled down by a further factor of $1/n$, so that the logits entering the softmax do not grow simply because the model got wider. That downscaling can be applied in either of two equivalent places — as an initialization variance of $1/n^2$, or as a $1/n$ multiplier on the output with ordinary initialization — and implementations differ in which they choose. Doing both is a common and silent error, since it applies the correction twice.\nThe payoff is the property the whole exercise was for. Under muP, the optimal learning rate stops depending on width. Tune it on a model of width two hundred and fifty-six, and the same value is optimal at width eight thousand. The curve of loss against learning rate, plotted for models of many widths, lines up its minimum at the same place instead of marching leftward. The same holds for several other dials — initialization scale, output multiplier, and to a workable approximation the settings of AdamW .\nFigure 1. Loss against learning rate, standard versus muP.\nFigure 2. Three classes of matrix, three scaling rules.\nA pinch of math: what is being held fixed The reasoning is easier to trust when the target is written down. Let $n$ be the width. For each layer we want three quantities to be of order one — neither growing nor shrinking — as $n$ becomes large:\n$$\\|h\\| \\sim \\Theta(1), \\qquad \\|\\Delta h\\| \\sim \\Theta(1), \\qquad \\|z\\| \\sim \\Theta(1)$$where $h$ is a layer\u0026rsquo;s activations, $\\Delta h$ is how much they move after one optimizer step, and $z$ are the final logits.\nThe first alone is what classical initialization achieves. The second is the new demand, and it is what forces the learning rate to scale. To see why, take a hidden matrix $W$ of shape $n \\times n$ acting on an activation $h$. An update $\\Delta W$ changes the output by $\\Delta W \\, h$, which is a sum of $n$ terms. If every entry of $\\Delta W$ is of the same typical size regardless of $n$ — which is what a width-independent learning rate gives you under Adam, since Adam\u0026rsquo;s update is normalized to a size set by the learning rate alone — then that sum grows like $n$, and the second condition fails badly. Scaling the learning rate for such matrices by $1/n$ cancels it exactly.\nThe third condition is why the output layer is special. Its input width grows, so without intervention the logits grow too, pushing the softmax toward saturation purely as a consequence of making the model wider — an effect that has nothing to do with the model being better and everything to do with the arithmetic.\nNone of this requires following the derivation. The point is that the scalings are not tuned; they are the unique solution to a set of constraints, which is why the transfer works reliably rather than approximately.\nFigure 3. What each condition is worth as width grows.\nWhat it changed in practice The procedure this enables is now standard at scale, and it is worth stating because it explains a great deal about how large models actually get built. Sweep the hyperparameters on a family of small models — cheap, fast, many runs. Verify that the optimum has stopped moving as width increases, which is the check that the parameterization is correctly implemented. Then transfer those values to the full-size run and train once. The cost of the sweep is a small percentage of the cost of the single large run, rather than a multiple of it.\nTwo honest caveats. The transfer is proven for width and is much less clean for depth, where the interaction with residual connections is more delicate and where the extensions that exist are newer and less settled. And implementing muP correctly is fiddly — it touches initialization, per-parameter-group learning rates, and an output multiplier, and getting any one of them wrong silently reverts you to standard behaviour while appearing to work. The recommended check is empirical rather than theoretical: train a family of increasingly wide models, plot loss against learning rate, and confirm the minima align. If they do not, something is wired incorrectly.\nWhat to carry forward Weights must start somewhere, and the scale of that start decides whether a signal survives eighty multiplications on the way up and whether a gradient survives them on the way down; the classical answer sets each weight\u0026rsquo;s variance to roughly one over its number of inputs, so that summing many terms neither amplifies nor shrinks. Transformers add two amendments: scale each block\u0026rsquo;s output down according to depth, because the residual stream accumulates and never subtracts, and start certain matrices at exactly zero so that a component begins as a no-op. What classical initialization does not give you is consistency across model sizes — it stabilizes the forward pass but says nothing about the size of an update, which grows with width, so the best learning rate drifts downward as models get wider and cannot be carried from a small experiment to a large run. Maximal update parameterization fixes this by deriving, rather than tuning, how initialization and learning rate must scale for each class of matrix so that activations, the change in activations per step, and the logits all stay a constant size as width grows. The consequence is that the optimal learning rate becomes width-independent, so hyperparameters can be swept on small cheap models and transferred intact to a model a hundred times larger — which is how frontier runs are configured, since nobody can afford to tune by trial and error at that scale. The transfer is solid for width, much less settled for depth, and easy to implement incorrectly in a way that fails silently, so the practical test is to confirm that the loss-versus-learning-rate minimum stops moving as width increases.\nThe figures in this article follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/articles/initialization/","summary":"\u003cp\u003eBefore the first gradient arrives, every weight in the model has to be set to something, and the something is drawn at random. This sounds like a detail to be got out of the way, and for a long time it was treated as one. It is not. The scale of those random draws determines whether a signal survives its journey up an eighty-block tower or dies on the way; it determines whether the gradient makes it back down; and — the part that turns a numerical nicety into an economic question — it determines whether a learning rate that worked on a small model still works on a large one. That last property has a name and a construction, and it is the reason nobody tunes hyperparameters on a frontier model directly. This piece is about how the starting point is chosen and why choosing it correctly lets you do your experiments somewhere cheap.\u003c/p\u003e","title":"Initialization and muP — Starting So That the Dials Transfer"},{"content":"One quantity keeps turning up across the training of language models, wearing a different job each time. It is the imitation metric in distillation , the leash in DPO and in reinforcement learning from human feedback, one of the damage measures in ablation , and the term that training actually minimizes once cross-entropy is decomposed. Four unrelated-looking uses, one formula. It is worth understanding on its own terms rather than meeting it four times as a piece of notation, particularly because it has a property that surprises everyone the first time — it is not symmetric, and which way round you write it changes what you are asking for. This piece is about what the quantity measures, why the asymmetry is a feature rather than a defect, and how to tell which direction you want.\nMeasuring a gap between beliefs Suppose two distributions over the same set of outcomes: $q$, which describes what actually happens, and $p$, which describes what you believe. How far apart are they?\nThe naive answers all fail. Comparing the most likely outcome under each throws away everything else. Summing the absolute differences treats a gap between 0.50 and 0.51 as equal in importance to a gap between 0.001 and 0.011 — but the second is a tenfold error and the first is a rounding difference. What is wanted is a comparison of ratios, and one that weights each outcome by how often it actually occurs.\nThe article on cross-entropy supplies the route. Surprise at an outcome of probability $p_i$ is $-\\log p_i$; the average surprise you actually experience, believing $p$ while reality follows $q$, is $-\\sum_i q_i \\log p_i$; and the least you could possibly experience, if your beliefs were exactly right, is $-\\sum_i q_i \\log q_i$. The gap between the two is the waste attributable to being wrong:\n$$D_{\\text{KL}}(q \\,\\|\\, p) \\;=\\; \\sum_i q_i \\log \\frac{q_i}{p_i}$$That is the Kullback–Leibler divergence, and the interpretation is exact rather than metaphorical. It is the number of extra bits per outcome you pay by encoding data from $q$ using a code designed for $p$. Zero when the two distributions coincide, positive otherwise, and never negative — a fact that follows from the concavity of the logarithm and can be taken on trust.\nThree properties are worth holding onto. It compares log-ratios, so it cares about relative rather than absolute differences. It weights each outcome by $q_i$, the probability of actually being there — outcomes that never occur under $q$ contribute nothing regardless of how wrong you are about them. And it is unbounded: if $q$ puts real probability on something $p$ considers impossible, the ratio blows up and the divergence is infinite.\nWhy it is not a distance Swap the arguments and you get a different number. $D_{\\text{KL}}(q \\| p) \\neq D_{\\text{KL}}(p \\| q)$, and often the two are wildly different. It also fails the triangle inequality. It is therefore not a metric, which is why the literature is careful to call it a divergence.\nThis looks like a defect and is in fact the most useful thing about it, because the two directions ask genuinely different questions.\nThe forward direction, $D_{\\text{KL}}(q \\| p)$, averages over where $q$ has mass. Every outcome that really happens contributes, weighted by how often. The penalty for putting too little probability on something real is severe — the log-ratio grows without bound as $p_i \\to 0$ — while putting probability on something that never happens costs nothing, since those terms are weighted by $q_i = 0$. Minimizing in this direction therefore produces a $p$ that covers everything $q$ does. Faced with a $q$ that has two separate peaks and a $p$ that can only manage one, the forward direction will smear $p$ across both, including the empty valley between them, rather than leave either peak uncovered. It is mass-covering, and its failure mode is a model that hedges, spreading probability over things that do not occur.\nThe reverse direction, $D_{\\text{KL}}(p \\| q)$, averages over where your own distribution has mass. Now the terms are weighted by $p_i$, so you are only penalized where you put probability. Placing mass where $q$ has none is catastrophic; declining to place mass somewhere $q$ has plenty costs nothing at all. Minimizing in this direction produces a $p$ that concentrates on one region and stays safely inside the support of $q$. Given the same two-peaked $q$, it picks a peak and commits. It is mode-seeking, and its failure mode is a model that ignores most of the possibilities.\nThe picture to keep is a blanket. The forward direction insists the blanket cover everything, and will accept a thin blanket over empty floor to manage it. The reverse direction insists the blanket only touch the bed, and will accept covering half the bed to guarantee it.\nFigure 1. Mass-covering and mode-seeking on the same target.\nWhich direction each use wants With that distinction, the four appearances sort themselves out.\nOrdinary training uses the forward direction, though it is rarely written that way. Minimizing cross-entropy against the empirical data is minimizing $D_{\\text{KL}}(\\text{data} \\| \\text{model})$, since the entropy of the data is a constant the model cannot affect. This is why language models are mass-covering — they assign some probability to nearly everything, and will happily hedge. It is the right choice for a model meant to represent the full range of what text can be, and it is also why sampling directly from a model gives less impressive text than truncated decoding does. The training objective asked for coverage; the reader wanted commitment.\nDistillation uses the forward direction with the teacher as $q$. The point is precisely to inherit the teacher\u0026rsquo;s full distribution, including the relative plausibility of the wrong answers, so mass-covering is exactly what is wanted.\nThe RL leash uses the reverse direction, with the policy as $p$ and the starting model as $q$ — and here the mode-seeking property is doing deliberate work. The penalty is enormous for the policy putting probability anywhere the reference model considered near-impossible, which is exactly the constraint desired: do not wander off into strange regions of language where the reward model\u0026rsquo;s judgement cannot be trusted. It is indifferent to the policy narrowing, which is also correct, since narrowing onto good behaviour is the entire point of the exercise. A forward-direction leash would have insisted the policy keep hedging like the base model, which would defeat the purpose.\nDPO does not compute a divergence at all, but its implicit reward is $\\beta \\log(\\pi/\\pi_{\\text{ref}})$ — the log-ratio that sits inside the sum. The constraint is built into the shape of the reward rather than added as a term, which is what that article means by the leash being structural.\nAblation uses it as a symmetric-in-spirit damage measure, asking how much the whole output distribution moved when a component was removed. Here neither direction is obviously right, which is why practitioners often report the symmetrized version instead.\nFigure 2. Which direction each use of KL wants.\nEstimating it in practice Two practical points, because the formula as written is often not what gets computed.\nOver a full vocabulary the sum is exact and cheap, since the model gives all the probabilities. This is the case in distillation and in ablation, and there is nothing to approximate.\nOver sequences it is intractable, because the sum runs over every string that could ever be generated. The RL leash is in this situation, and the usual approach is to estimate from samples: draw sequences from the policy and average $\\log(\\pi/\\pi_{\\text{ref}})$ over them. This is unbiased but noisy, and it can go negative on any particular batch even though the true value cannot, which is disconcerting when it appears on a dashboard. A better estimator adds a correction term that is zero in expectation but cancels much of the variance, giving something that stays non-negative sample by sample. It is a small implementation detail with a visible effect on how stable a training run looks.\nThe other point concerns the infinity. If the model assigns exactly zero to something the reference assigns real probability, the divergence is infinite and the gradient is meaningless. In practice a softmax never outputs exactly zero, so this rarely bites during training — but it bites immediately if the distribution has been truncated before comparison, which is a genuine trap when evaluating models under their sampling settings rather than raw.\nWhat to carry forward KL divergence measures the gap between two distributions as the extra bits per outcome you pay for believing $p$ when reality follows $q$ — a comparison of log-ratios, weighted by how often each outcome actually occurs, zero when the distributions agree and never negative. It is not symmetric, and the asymmetry is the point. The forward direction, averaging over where the truth has mass, punishes leaving anything real uncovered and ignores probability spent on things that never happen, so it is mass-covering and produces models that hedge; the reverse direction, averaging over where your own distribution has mass, punishes only where you have committed, so it is mode-seeking and produces models that pick a region and stay inside it. Ordinary language-model training and distillation both use the forward direction, which is why models hedge by default and why truncated decoding is needed to get committed text out of them. The reinforcement-learning leash uses the reverse direction deliberately, because the goal is to forbid wandering into regions the reference model considered implausible while permitting the policy to narrow. Over a vocabulary the sum is exact and cheap; over sequences it must be estimated from samples, where a variance-reduced estimator avoids the disconcerting negative values a naive one produces. And the quantity is unbounded, going infinite the moment one distribution assigns real probability to something the other rules out entirely.\nThe figures in this article follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/articles/kl-divergence/","summary":"\u003cp\u003eOne quantity keeps turning up across the training of language models, wearing a different job each time. It is the imitation metric in \u003ca href=\"/articles/distillation/\"\u003edistillation\u003c/a\u003e\n, the leash in \u003ca href=\"/articles/dpo/\"\u003eDPO\u003c/a\u003e\n and in reinforcement learning from human feedback, one of the damage measures in \u003ca href=\"/articles/ablation/\"\u003eablation\u003c/a\u003e\n, and the term that training actually minimizes once \u003ca href=\"/articles/cross-entropy/\"\u003ecross-entropy\u003c/a\u003e\n is decomposed. Four unrelated-looking uses, one formula. It is worth understanding on its own terms rather than meeting it four times as a piece of notation, particularly because it has a property that surprises everyone the first time — it is not symmetric, and which way round you write it changes what you are asking for. This piece is about what the quantity measures, why the asymmetry is a feature rather than a defect, and how to tell which direction you want.\u003c/p\u003e","title":"KL Divergence — The Distance Between Two Sets of Odds"},{"content":"The generation loop takes one line to state: the model reads everything so far, produces a distribution over the vocabulary, one token is chosen and appended, and the whole thing runs again. Described that way it sounds tidy, and it hides something absurd. Taken literally, producing the thousandth word of an answer would mean pushing all nine hundred and ninety-nine earlier words back up through every block of the tower, from scratch, to compute representations the model had already computed nine hundred and ninety-nine times before. No system does this, and the thing that saves it is the KV cache — a running set of notes that turns a re-reading problem into a memory problem. This piece is about what is in those notes, why they consist of keys and values and nothing else, and why the memory problem they create now governs almost everything about what it costs to run a model.\nThe waste in the obvious loop Start by pricing the naive version, because the number is what motivates everything else.\nTo produce one new token, the model must run a forward pass. If that pass has to process the entire sequence, then generating a response of $N$ tokens costs a pass over one token, then over two, then over three, and so on — which totals on the order of $N^2/2$ token-positions of work. A two-thousand-word answer would require processing something like two million token-positions rather than two thousand. The model would spend a thousand times more effort re-deriving what it already knew than on the words it was actually producing.\nNow the crucial observation, and it is a property of the architecture rather than an optimization anyone invented. In a decoder, attention is causally masked: every position may look backwards and none may look forwards. So the representation the model builds for token five depends on tokens one through five and nothing else. When token six arrives, token five\u0026rsquo;s representation does not change. It cannot change, because token five was never permitted to see token six in the first place.\nThat is what licenses caching. The recomputation is not merely expensive; it is provably redundant, producing bit-for-bit identical results every time. Whatever the model worked out about the earlier part of the sequence remains true, and the only genuinely new work is whatever the newest token requires. Caching here is not an approximation and costs nothing in quality — it is the removal of work that was pure waste.\nWhy keys and values, and not queries So the model should save its work on earlier tokens. Save what, exactly? The answer is more specific than \u0026ldquo;the earlier computation,\u0026rdquo; and the specificity is the whole reason for the name.\nRecall what each token contributes inside an attention head. From its own vector it produces three things: a query, expressing what it is looking for; a key, advertising what it offers to anyone looking; and a value, the content it hands over to whoever attends to it. Attention pairs each token\u0026rsquo;s query against every earlier token\u0026rsquo;s key, and mixes the corresponding values in proportion.\nNow ask, for each of these three, whether it is ever needed again after its own position has been processed.\nA key is. When token six arrives, its query is compared against the keys of tokens one through five — keys that were produced when those tokens were processed, and that will be consulted again by token seven, and token eight, and every token thereafter. The same is true of values: token five\u0026rsquo;s value will be mixed into the output of every future position that attends to it. Keys and values are read by other tokens, indefinitely into the future.\nA query is not. Token five\u0026rsquo;s query was used exactly once, at position five, to gather from the past. Nothing in the future ever consults it, because attention flows backwards only. The moment position five is finished, its query is dead.\nThat asymmetry is the entire design. Cache the keys and the values; throw the queries away. Hence the name, and hence the shape: two tensors per attention layer, holding one key vector and one value vector for every token processed so far.\nWith those notes in hand the generation step becomes narrow. The new token enters at the bottom of the tower as a single vector. At each layer it produces its own query, key, and value; its query attends over the cached keys and values plus its own; the result goes through the rest of the block; and its freshly minted key and value are appended to the cache for that layer before the vector moves up. What travels through the tower is a column exactly one token wide, consulting a widening archive at every level. The cost per generated token no longer grows with the length of what came before — except in the attention step itself, which must still scan the whole cache, and which is cheap by comparison with everything else.\nFigure 1. Recomputing the tower versus consulting the notes.\nA pinch of math: how big the notes get The cache is not free, and its size is worth being able to estimate on the back of an envelope, because it explains more about the economics of serving models than any other single number.\nFor each token, each layer stores one key and one value per attention head, and each of those is a vector as wide as a head. So:\n$$\\text{bytes per token} = 2 \\times L \\times n_{kv} \\times d_{\\text{head}} \\times b$$The $2$ is for keys and values, $L$ is the number of layers, $n_{kv}$ the number of key-value heads, $d_{\\text{head}}$ the width of each, and $b$ the bytes per number.\nPut a real model in. Thirty-two layers, thirty-two heads of width one hundred and twenty-eight, stored in half precision at two bytes each. That gives $2 \\times 32 \\times 32 \\times 128 \\times 2$, which is a little over half a megabyte — per token.\nHalf a megabyte does not sound like much until you multiply. An eight-thousand-token conversation carries a four-gigabyte cache. A thirty-two-thousand-token one carries sixteen gigabytes, which for a seven-billion-parameter model in half precision is more memory than the model\u0026rsquo;s own weights. And these figures are for a single conversation. Serve thirty-two users at eight thousand tokens each and the caches alone want a hundred and twenty-eight gigabytes, against fourteen for the weights they are all sharing.\nThat last comparison is the one to hold onto, and it has a clean statement. The weights are a library: one copy, consulted by everybody. The cache is a desk: every reader needs their own, and it keeps growing as they work. Adding readers costs you nothing in library, and a desk apiece.\nFigure 2. Library and desks — weights against caches.\nTwo regimes: filling the cache and using it Serving a request divides into two phases that look superficially similar and behave like different machines.\nPrefill is what happens to the prompt. The whole thing is available at once, so every position can be processed in parallel — the same computation the model performs during training. The matrices involved are large and dense, the hardware is doing what it is built for, and the phase is limited by raw arithmetic. It is compute-bound. Its output, besides the first generated token, is a fully populated cache.\nDecode is everything after: one token at a time, each depending on the one before, with no way to parallelize across the sequence. And here the arithmetic goes strange. To produce a single token, every weight in the model must be read out of memory — all fourteen gigabytes of a seven-billion-parameter model — in order to perform roughly fourteen billion multiply-adds. That is about one arithmetic operation for every byte moved. A modern accelerator can perform several hundred operations in the time it takes to move one byte from its memory. So during decode the arithmetic units sit idle well over ninety-nine percent of the time, waiting for weights to arrive. The phase is memory-bandwidth-bound, and how fast a model generates text is, to a first approximation, a question of how fast its weights can be streamed rather than how fast the chip can compute.\nThis split explains why serving systems report two separate numbers — time to the first token, and time per token thereafter. They are governed by different physics, respond to different optimizations, and trade against each other. A long prompt with a short answer is a prefill problem; a short prompt with a long answer is a bandwidth problem.\nFigure 3. Prefill is compute-bound, decode is bandwidth-bound.\nWhy batching is the whole business model Follow the bandwidth argument one step further and something important falls out.\nIf decode is spending nearly all its time reading weights and almost none doing arithmetic, then the arithmetic is free. Process sixty-four sequences at once and you read each weight matrix once, then use it sixty-four times instead of once. The reading — the expensive part — is amortized across the whole batch. Sixty-four users can be served in barely more wall-clock time than one.\nThis is not a marginal efficiency. It is the reason serving a model costs cents rather than dollars, and it is why every inference system batches as aggressively as it possibly can. But notice what limits it. The weights amortize; the cache does not. Each additional sequence in the batch brings its own desk, and the desks are what fill the machine. The size of the KV cache therefore sets the batch size, the batch size sets the throughput, and the throughput sets the price per token. Every megabyte shaved off the cache converts more or less directly into another concurrent user, which is why the allocator handing out that memory — the subject of paged attention — turned out to matter as much as the cache itself.\nThere is a scheduling refinement worth knowing, because it follows from the same asymmetry. If a batch is assembled and run to completion together, every sequence is held hostage by the longest one, and finished slots sit idle. Continuous batching instead schedules at the level of individual decode steps, retiring a sequence the moment it ends and admitting a waiting request into the freed slot on the very next iteration. The batch is a revolving door rather than a bus, and in practice it is worth several times the throughput of the naive arrangement.\nMaking the notes smaller Given that the cache determines the economics, a great deal of ingenuity has gone into shrinking it. The formula above has four factors that can be attacked, and all four have been.\nThe most widely deployed attack is on $n_{kv}$: let several query heads share one set of keys and values. Since queries are not cached, you can keep all thirty-two query heads — preserving most of the model\u0026rsquo;s ability to attend in different ways — while storing only eight sets of keys and values, or in the extreme only one. Eight groups instead of thirty-two divides the cache by four, for a quality cost small enough that essentially every recent model does it. It is a remarkably good trade, and it works precisely because the thing being shared is the part that has to be stored.\nA subtler attack compresses the keys and values jointly. Rather than storing them directly, project them down into a much smaller shared vector, cache that, and reconstruct what is needed on the fly — the same observation about big matrices that underpins LoRA , that the useful content of a large object often occupies far fewer dimensions than the object has. Applied to the cache it can remove the great majority of it, and it is the approach behind the long contexts of some recent large models.\nA third attack is on $b$, the bytes per number: store the cache in eight or even four bits rather than sixteen. This is quantization pointed at the cache instead of the weights, and it inherits the same crucial split — keep the stored form small, expand it back to full precision for the arithmetic. Keys turn out to tolerate this less gracefully than values, so the two are often quantized to different depths.\nThe last attack is on the sequence length itself, and it is the only one that is genuinely lossy. If old tokens can be evicted, the cache stops growing. Sliding-window attention keeps only the most recent few thousand positions; other schemes evict tokens that have attracted little attention historically. A surprising practical finding is that the very first tokens of a sequence must be kept regardless of content, because attention deposits a great deal of leftover probability mass on them, and discarding them destroys the model\u0026rsquo;s calibration in a way that discarding the middle does not. Whatever the policy, evicting means the model can no longer see what was evicted, so this is a change to behaviour rather than an optimization — the honest framing is that it trades recall of the distant past for the ability to run at all.\nManaging the notes you keep Alongside making the cache smaller sits the question of storing it well, and this turns out to be where a large fraction of real-world serving performance was hiding.\nThe obvious implementation allocates, for each request, one contiguous block of memory big enough for the longest response it might produce. It is also disastrous. Most requests finish far short of their limit, so most of every reservation is never used, and the leftover gaps are the wrong shape for the next request. Systems built this way commonly wasted more than half of the memory they had set aside.\nThe fix borrows an idea from operating systems, which solved this problem for physical memory decades ago. Break the cache into small fixed-size blocks — say sixteen tokens each — scatter them anywhere in memory, and keep a table mapping each sequence\u0026rsquo;s logical positions to wherever its blocks physically live. A sequence grows by claiming one more block when it needs one and no sooner. Fragmentation collapses to at most a few unused slots at the end of a single block, memory utilization goes from roughly half to nearly all of what is available, and since the cache determines batch size, that translates straight into throughput.\nThe page table then enables something better. If two requests begin with identical text — the same system prompt, the same few-shot examples, the same document being asked about repeatedly — the cache blocks for that shared prefix are also identical, and both requests can simply point at the same blocks. Nothing is copied until one of them diverges. In production this is enormous: a two-thousand-token system prompt prefixed to every request is prefilled once and shared by everyone thereafter. It is also the mechanism behind the prompt-caching features that model providers now expose and bill at a steep discount — the discount is real, because the work genuinely was not done again.\nWhen memory still runs short, the remaining options are to move a cache out to system RAM and bring it back when the conversation resumes, or simply to discard it and recompute the prefill later. Which is cheaper is a live calculation rather than an obvious call, since prefill is fast and the link to system memory is slow, and a long-idle conversation is often more cheaply re-read than retrieved.\nWhat this explains about everything else The cache is one mechanism, but it sits underneath a surprising number of things that otherwise look unrelated.\nLong context is a memory problem, not an intelligence problem. When a model is advertised as handling a million tokens, most of the engineering behind that claim is cache management — sharing, compression, eviction, paging — rather than anything about the model\u0026rsquo;s ability to reason over long text. The two are routinely conflated, and separating them makes a lot of product claims easier to read.\nSpeculative decoding works because of the bandwidth argument, and only because of it. A small fast model drafts several tokens; the large model then checks all of them in a single forward pass and accepts the longest correct prefix. This is a gain rather than a wash only because verifying five tokens costs the large model almost exactly what generating one costs — the weights are read once either way, and the extra arithmetic is free in a regime that had compute to spare. It is a way of converting idle arithmetic units into tokens, and the guarantee is exact: the accepted output is distributed identically to what the large model would have produced alone.\nIt sharpens the mixture-of-experts bargain. A sparse model already inverts the usual picture by spending memory to save computation. In a decode phase that was bandwidth-bound to begin with, the saved computation is worth less than it looks and the extra memory hurts more, since it competes with the caches for the space that determines batch size. Sparse models are still a good trade — but the case rests on serving them at large scale across many machines, where the routing spreads work that would otherwise be duplicated, rather than on the raw arithmetic saving.\nAnd it is why thinking is expensive. A reasoning model that deliberates for ten thousand tokens before answering has built a ten-thousand-token cache, occupying a desk that whole time, before it says its first word to you. Test-time compute is paid for twice: once in generated tokens, and once in the memory occupied while generating them, which is memory not available to serve anyone else.\nWhat to carry forward Generating text one token at a time would mean re-deriving every earlier token\u0026rsquo;s representation on every step, and causal masking guarantees those re-derivations would be identical — so they are computed once and kept. What is kept is specifically the keys and values of each attention layer, because those are consulted by every future position, while a query is used once at its own position and then never again; that asymmetry is the whole reason for the name. The saved notes are large: roughly half a megabyte per token for a mid-sized model, which means a long conversation\u0026rsquo;s cache can exceed the model\u0026rsquo;s own weights, and unlike the weights it is not shared — the weights are a library, the cache is a desk per reader. Serving therefore splits into a compute-bound prefill over the prompt and a memory-bandwidth-bound decode that reads every weight to produce a single token, using well under one percent of the available arithmetic. That idle arithmetic is what makes batching nearly free and speculative decoding profitable, and it is what makes cache size the binding constraint: the cache sets the batch size, the batch size sets throughput, and throughput sets the price. Hence the effort spent shrinking it — sharing key-value heads across query heads, compressing keys and values into a low-rank latent, quantizing the stored form, and, at the cost of actual capability, evicting old tokens — and the equal effort spent managing it, with fixed-size paged blocks to kill fragmentation and shared blocks to let a common prefix be prefilled once for everybody, which is exactly what a prompt-caching discount is selling. Most of what sounds like a claim about long-context intelligence is, underneath, a claim about how well someone is managing this one data structure.\nThe figures in this article follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/articles/kv-cache/","summary":"\u003cp\u003eThe generation loop takes one line to state: the model reads everything so far, produces a distribution over the vocabulary, one token is chosen and appended, and the whole thing runs again. Described that way it sounds tidy, and it hides something absurd. Taken literally, producing the thousandth word of an answer would mean pushing all nine hundred and ninety-nine earlier words back up through every block of the tower, from scratch, to compute representations the model had already computed nine hundred and ninety-nine times before. No system does this, and the thing that saves it is the \u003cem\u003eKV cache\u003c/em\u003e — a running set of notes that turns a re-reading problem into a memory problem. This piece is about what is in those notes, why they consist of keys and values and nothing else, and why the memory problem they create now governs almost everything about what it costs to run a model.\u003c/p\u003e","title":"KV Caching — Reading Each Word Once, and Carrying It Forever"},{"content":"The limit on human preference data is usually presented as a matter of money: annotation is expensive, budgets are finite, so the field went shopping for something cheaper. That framing gets the constraint wrong. What limits preference data is not its price but the fact that somebody has to read two long responses attentively enough to have a defensible opinion about which is better, and attention of that kind does not scale and cannot be bought in bulk. A second limit sits behind it: an annotator cannot demonstrate a good answer to a question they cannot answer, so on difficult material the ceiling is competence rather than diligence. The field\u0026rsquo;s answer was to hand the comparison to a model. This piece is about what that substitution leaves untouched, what it makes newly legible, and the moment its errors stop cancelling.\nThe constraint is arithmetic, not sentiment Put numbers on it, loosely. A preference dataset that meaningfully moves a model\u0026rsquo;s behaviour wants comparisons in the tens or hundreds of thousands. Each is a prompt and two responses of a few hundred words, and a person who skims them produces a label worse than no label, because noise in preference data is indistinguishable from taste. Budget a few minutes per pair and the arithmetic turns unpleasant: a hundred thousand comparisons is thousands of hours of undivided attention, the one input in the training stack whose unit cost has never fallen.\nThen multiply, because helpfulness is only the first axis. Tone is another, and it contradicts helpfulness at the margins. Format, refusal behaviour , factual care, ambiguous requests, the right quantity of hedging — each is a separate property, wanting comparisons drawn from prompts that exercise it. The demand is a product rather than a sum. The bottleneck is not enthusiasm, and not really money either. It is hours.\nThe move that follows is unromantic. Show a capable model the same prompt and the same two responses, with a written statement of what to prefer, and record whichever it picks as the label. Nothing downstream changes. The pairs feed a reward model exactly as human pairs did, or go straight into direct preference optimization , and the policy optimization consuming the result cannot tell where they came from. It is easy to mistake this for a new algorithm. It is a change of data source. Everything already true of preference learning — the reward is a proxy, the proxy gets gamed, the leash to the reference model is load-bearing — remains true, with one fewer person in the room.\nFigure 1. The same pipeline, one fewer person in the room.\nThe written principles are the interesting part This is where the substitution earns more than a line in a budget, and the reason is legibility rather than cost. Human preference data encodes its criteria implicitly. What the annotators were rewarding lives distributed across a hundred thousand judgements, each a single bit, none with a reason attached. To find out what such a dataset asked for, you fit something to it and observe what the result favours. That is an experiment, not a reading.\nA written statement of what to prefer is a different kind of object. It is explicit, it sits in version control, and it can be argued with by people who never saw a label. You can read the line saying brevity is preferred where no detail is lost, decide it is producing answers that omit caveats, change that line, relabel, retrain, and compare. That is a real gain, and it deserves stating before it is qualified.\nHere is the qualification. What has become legible is the instruction, not the behaviour. The document is an input to the judge\u0026rsquo;s reading of a pair, and that reading is a forward pass through a network with habits of its own. The gap between what the principles say and what the model does is what the procedure both closes and fails to close: closed enough that the document is worth writing, open enough that reading it tells you what was asked for rather than what was learned. An auditable specification attached to unaudited behaviour beats no specification. It is not a guarantee, though it is occasionally taken for one.\nJudging is easier than producing, and only up to a point That the labels carry information rests on an asymmetry worth stating carefully. Deciding which of two answers is better is strictly easier than writing the better answer. A reader who could not have derived a proof can still notice that one attempt skips a case. So a model supplies useful signal about outputs it could not have produced — the same asymmetry that makes checking cheaper than generating in verifiable rewards , where a program that could never write the code under test can still run it, and the reason sampling many candidates and picking among them beats spending the same compute on one.\nThe asymmetry is real and bounded, and the boundary sits somewhere inconvenient. It holds where the flaw is visible in the response: a contradiction, an omission, an evasion, a mess. It collapses wherever recognizing the better answer needs knowledge the judge lacks. A fabricated citation with a plausible author and year reads as scholarship to a judge that has not read the literature, and the confident wrong answer beats the hedged right one on every axis available — much of the mechanism behind hallucination surviving training. That is the territory where the human ceiling hurt most, and the model ceiling sits in the same place.\nAn unsettled argument rests on top of this, better named than resolved. One reading treats model feedback as a new source of signal: the judge does work at label time that nobody did before. The other treats it as distillation — the judge\u0026rsquo;s preferences amortized into a second model at volume, the written principles serving as a prompt rather than as information. Which reading applies depends on the arrangement. Where the judge is much stronger than the model being trained, distillation explains most of what happens and predicts the ceiling correctly. Where the judge is the same model, or weaker, distillation explains nothing and the asymmetry does all the work. Both are in use, which is much of why the argument persists.\nFigure 2. Where judging stops being easier than producing.\nA pinch of math: what averaging annotators was buying The cleanest formal statement of the cost is a decomposition, and it explains what a crowd was for. Write one labeller\u0026rsquo;s error on one comparison as a sum of two parts:\n$$\\varepsilon_i = b + v_i$$Here $b$ is the bias shared by everyone labelling, a systematic lean toward longer responses, say, and $v_i$ is idiosyncratic noise, independent across labellers, with mean zero and variance $\\sigma^2$. Collect $n$ labels for one comparison and average them. The shared part survives untouched, while the idiosyncratic part becomes $\\frac{1}{n}\\sum_i v_i$, leaving\n$$\\mathbb{E}\\big[\\bar{\\varepsilon}^{\\,2}\\big] = b^2 + \\frac{\\sigma^2}{n}$$The second term shrinks as $1/n$, so its typical size shrinks as $1/\\sqrt{n}$, and can be pushed as low as the budget allows. The first term does not move. Total error does not tend to zero as labellers are added; it tends to $b^2$.\nRead both arrangements against that. A pool of human annotators was expensive because it was buying reductions in $v$, and it worked: human $v$ is large while human $b$ is small and pulls in several directions at once — one rater\u0026rsquo;s systematic lean is, across a diverse pool, part of somebody else\u0026rsquo;s noise. A single model inverts both halves. Reducing $v$ becomes nearly free, since the judge can be sampled repeatedly or run cool. And $b$ becomes irreducible, because there is one labeller and every comparison inherits the same lean. The rounding error is now the whole of the error.\nFigure 3. Error floors at the judge\u0026rsquo;s bias, not zero.\nWhat the substitution costs Three consequences follow, and together they make this a trade rather than a saving.\nThe errors correlate. Ten annotators who each misjudge a different tenth of the pairs produce scattered, partly self-cancelling noise. One model that misjudges a particular kind of pair misjudges every instance of that kind, in the same direction, a million times. Its blind spots are not averaged away; they are stamped uniformly into the reward signal, indistinguishable from preferences and learned as such. The failure is not that the model is a worse judge than a person, which it often is not. The failure is that its mistakes arrive in perfect agreement with each other.\nThe judge has documented biases. It prefers longer answers. It prefers confident phrasing to hedged phrasing. It prefers whichever option came first, so pairs must be run in both orders and averaged. It prefers text resembling its own style, which makes a judge from one candidate\u0026rsquo;s own family a partisan rather than an arbiter. That is the list that turns up whenever models are used as graders, as the article on evaluation sets out, and it is the same list because it is the same act. An evaluation with known distortions can be corrected for; a training signal with known distortions becomes the thing optimized. The bias stops being measurement error and becomes the objective.\nIt is a closed loop. A model producing the data that trains the next model has no external reference, and preferences nobody wrote down compound across rounds into behaviour nobody chose — a register that drifts, a house style that hardens, a category of question that stops being answered. Detecting that means reading outputs and judging them, which is the human evaluation the arrangement was built to reduce. The saving is partly a transfer rather than an elimination: hours move out of labelling and into auditing, at a better exchange rate but not a free one.\nWhich is why hybrids dominate practice, and why that is the honest empirical position rather than a compromise. Model labels cover volume, supplying the comparisons no budget would otherwise reach. Human labels are kept for calibration — a sample scored both ways, to see where the judge agrees with people and where it does not — and for the cases nobody will let a model decide, usually the safety boundary and whatever the last audit found going wrong.\nWhat to carry forward Preference training wants a great many comparisons, each requiring a person to read two responses with real attention, and attention of that kind has never got cheaper; multiply by every axis worth training — helpfulness, tone, format, refusal, factual care — and the demand becomes a product rather than a sum. Handing the comparison to a capable model, guided by a written statement of what to prefer, relieves that without altering anything downstream: the labels feed a reward model or a direct preference objective exactly as human labels did, so this is a change of data source, and every existing property of preference learning survives intact. The written principles are the real gain, because criteria that previously existed only as a distribution over thousands of unexplained judgements become a document you can read, dispute, and amend one line at a time — though what is legible is the instruction rather than the behaviour, and that distance is not closed by writing the instruction more carefully. The labels carry information because judging is easier than producing, the asymmetry that makes checking cheaper than generating elsewhere; it is bounded, failing wherever recognizing the better answer needs knowledge the judge lacks, which is where help was most wanted. The costs are structural. Independent human errors partly cancel and one judge\u0026rsquo;s errors do not, so total error tends to a floor set by the judge\u0026rsquo;s bias rather than toward zero. Its documented leanings — length, confidence, position, its own style — become the objective. And a model generating the data for the next model can drift somewhere nobody chose, detectable only by the human reading the arrangement was meant to avoid. What is bought is volume. What is spent is the diversity of error that made aggregation work at all.\nThe figures in this article follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/articles/ai-feedback/","summary":"\u003cp\u003eThe limit on human preference data is usually presented as a matter of money: annotation is expensive, budgets are finite, so the field went shopping for something cheaper. That framing gets the constraint wrong. What limits preference data is not its price but the fact that somebody has to read two long responses attentively enough to have a defensible opinion about which is better, and attention of that kind does not scale and cannot be bought in bulk. A second limit sits behind it: an annotator cannot demonstrate a good answer to a question they cannot answer, so on difficult material the ceiling is competence rather than diligence. The field\u0026rsquo;s answer was to hand the comparison to a model. This piece is about what that substitution leaves untouched, what it makes newly legible, and the moment its errors stop cancelling.\u003c/p\u003e","title":"Learning From AI Feedback — Replacing the Annotator With the Model"},{"content":"The learning rate is usually introduced as a number: one small constant, multiplied into every gradient and then left alone. Almost no large model has ever been trained that way. In practice it is a function of the step count — near zero for the opening few thousand steps, rising to a peak, then falling for the rest of the run — and the shape of that function is treated as somewhere between a solved problem and a superstition. It is neither. The opening ramp exists for a specific and slightly embarrassing reason to do with the optimizer\u0026rsquo;s own bookkeeping, and the conventional curve imposes a constraint so awkward that an entire alternative arrangement was invented to escape it. This piece is about why the rate has to move at all, what the two halves of the schedule are separately doing, and what the standard shape quietly costs.\nOne step size cannot serve a whole run The rate has to move because the run is not one problem. It is at least two, and they want opposite things.\nEarly on, almost any direction is downhill. The weights are random draws, the model is producing noise, and the gradients arriving from successive batches are large and in broad agreement, because every example is objecting to the same gross errors — the output distribution is flat where it ought to be sharply peaked, and correcting that is a consensus project. There is little disagreement to average away and a great deal of ground to cover, so a long step is valuable, though the optimizer\u0026rsquo;s own running estimates have barely been assembled.\nLate on, what remains is disagreement. The obvious structure has been absorbed, the gradients are small, and consecutive batches now argue about what to do next, since what is left to learn is precisely the part on which examples do not agree. A long step here covers no ground. It responds at full volume to whichever few hundred thousand tokens happened to arrive, then does it again in the opposite direction.\nA single number is either too timid for the first regime or too loud for the second, and the schedule is the admission that no compromise value is good in either.\nThe peak value has no theory behind it either, and two couplings catch people out. It depends on the batch size , since a cleaner gradient supports a bolder step, and the working rule is to scale the rate with the square root of the batch — approximate, and valid only up to a ceiling set by the curvature of the landscape rather than by the data. It depends also on the width of the model, which is why a rate tuned on a small proxy does not transfer under ordinary initialization ; making it transfer is exactly what muP-style parameterization is for.\nFigure 1. What the run wants early and what it wants late.\nWarm-up buys time for a badly estimated denominator The folklore explanation for warm-up is that the model is fragile at the start and should be handled gently. That is not wrong. It is also not the mechanism.\nAn adaptive optimizer such as AdamW does not apply the gradient directly. It divides the smoothed gradient by a running estimate of recent gradient magnitude, so that every parameter steps in units of its own typical behaviour. The denominator is the whole trick. At step ten it is an average over ten samples, and an estimate built from that little data is not merely imprecise; it is occasionally wildly wrong in the dangerous direction, too small. A confident step divided by a badly underestimated denominator is an enormous step taken on no evidence. That is how a run dies in its first thousand steps, quietly, leaving a curve that looks as though learning never started rather than one that broke.\nRamping the rate from near zero over a few thousand steps buys time for the estimate to accumulate enough samples to be trusted, at the cost of steps deliberately smaller than would have worked. It is insurance, paid for in data.\nThis is also why bias correction exists, and the two are easily confused. The running averages start at zero, so for the opening steps they are dragged toward zero and understate both the gradient and its magnitude; the correction inflates them back to honest size. That fixes the bias. It does nothing about the variance, which is what warm-up is for. A correctly bias-corrected optimizer with no warm-up still takes a catastrophic step the first time its denominator comes out small, so anyone who removes the ramp on the grounds that bias correction already handles the opening steps will meet a loss spike within the hour.\nFigure 2. How a badly estimated denominator kills a run.\nDecay converts bouncing into settling What the decay buys is best described geometrically. Every step of gradient descent is the true descent direction plus a noise term inherited from sampling a finite batch, and both scale with the rate. At a large rate the noise is large enough that the parameters circulate around the bottom of a region at a fixed radius rather than descending into it, like tea leaves in a cup still being stirred. Lowering the rate shrinks the radius. Circulation becomes descent, and the loss drops for reasons that have nothing to do with new information and everything to do with the parameters finally being allowed to arrive.\nThat is the honest account of the sharp late-run improvement people see when the rate anneals. It is not the model suddenly learning. It is the model stopping.\nThe conventional curve is cosine decay, from the peak down to a tenth of it or to zero across the whole run, and its standing deserves scrutiny. Linear decay works about as well; inverse-square-root nearly as well; at matched budgets the gaps between them are small enough to be swamped by other choices. What matters is that the rate spends a long stretch high and finishes very low, not the route between. There is no derivation of the shape. Cosine is a convention that works, kept because it is what everyone else uses, and treating it as an optimum is the same error as treating any long-standing default as a result.\nDecaying the rate also weakens the weight decay, and almost nobody accounts for it. Decoupled decay pulls every weight toward zero by a fraction proportional to the current rate, so as the rate falls the tether slackens in exact proportion, and by the end of a cosine run the regularization operates at a tenth of its nominal strength or less. The schedule anneals two things while appearing to anneal one.\nA pinch of math: where the total step count hides Write $\\eta_t$ for the rate at step $t$. The warm-up is a straight line, $\\eta_t = \\eta_{\\max}\\, t / T_w$ for $t \u003c T_w$, and the cosine phase that follows is\n$$\\eta_t = \\eta_{\\min} + \\tfrac{1}{2}(\\eta_{\\max}-\\eta_{\\min})\\left(1 + \\cos\\left(\\frac{\\pi t}{T}\\right)\\right)$$At $t = 0$ the cosine is $1$ and the rate is $\\eta_{\\max}$; at $t = T$ it is $-1$ and the rate is $\\eta_{\\min}$. The derivative is near zero at both ends and largest in between, so the curve lingers at the top, falls quickly through the middle, and lingers again at the bottom — the property usually credited for its success.\nThe load-bearing symbol is $T$. It is not a stopping condition applied to a curve defined independently of it; it sits inside the argument of the cosine, so the rate at step fifty thousand of a run planned for one hundred thousand steps is not the rate at step fifty thousand of a run planned for two hundred thousand. Doubling the intended length does not extend the schedule. It redraws every point on it. That is the whole practical problem in one line: to evaluate $\\eta_t$ at all, you must already have decided when the run ends.\nThe plan is fixed before the run begins Because the total step count lives inside the formula, a cosine run is one indivisible object. It cannot be extended, since extending it would have required a different curve from the start, and cannot be usefully resumed at a longer horizon for the same reason. Every checkpoint before the end is not a finished model trained for less time but a model caught mid-anneal, still circulating at a rate chosen for a run that has not finished, and therefore worse than a model properly cooled down at that budget. For anyone who wanted a curve of loss against training duration, that is expensive: every point on it is a separate run.\nThe fix is to separate the high phase from the descent. Hold the rate flat for the great majority of the run, then decay it sharply at the very end over a few percent of the steps. This constant-then-cooldown arrangement reaches loss comparable to a well-tuned cosine and removes the dependence on knowing the total in advance, because a flat phase is indifferent to when it stops. Train as long as the budget allows, stop, cool down over a short tail, and the result is a finished model rather than an interrupted one. This is what makes the scaling-law practice of reading many points off one run possible, and a fair illustration of how far a scheduling convention can constrain the experiments people are able to imagine.\nWhat no schedule does is react to the run. It is an open-loop plan, computed from the step index and nothing else, indifferent to whether the loss is falling nicely, has flattened, or has just spiked by two nats. The optimizer adapts per parameter; the global rate does not adapt at all. Alternatives exist — rates derived from gradient statistics, methods that dispense with a horizon, controllers that watch the loss — and they have not displaced the fixed schedule despite being competitive in narrow comparisons. The reason is not that they fail. It is that a fixed schedule is predictable, so it can be planned in advance and blamed or exonerated afterwards, and predictability is worth a great deal when a run costs millions and cannot be repeated.\nFigure 3. Warm-up, cosine, and the flat-then-cooldown alternative.\nWhat to carry forward A learning rate is a function, not a constant, because a run makes contradictory demands: early, the model is producing noise, gradients are large and in broad agreement, and long steps are safe; late, gradients are small and mutually contradictory, and the useful move is to settle. Warm-up handles the opening, for a sharper reason than fragility — an adaptive optimizer divides by a running estimate of gradient magnitude, built from almost no samples at first, and a confident step divided by a badly underestimated denominator is how a run dies in its first thousand steps; ramping from near zero buys time for it to be trusted, a different complaint from the systematic understatement bias correction fixes, and conflating the two loses runs. Decay converts circulation into descent, since a high rate keeps the parameters orbiting a basin rather than entering it; the late fall in loss is arrival, not insight. Cosine is the default, its shape matters less than its reputation, and no derivation stands behind it. Its real cost is that the total step count sits inside the formula: the run cannot be extended or resumed, and every intermediate checkpoint is mid-anneal rather than finished, which is why holding the rate flat and cooling sharply at the end has displaced it wherever the stopping point is not yet chosen. Two couplings stay easy to miss — the peak scales roughly with the square root of the batch and will not cross model widths without a parameterization built for it, and decoupled weight decay is proportional to the current rate, so annealing one anneals the other. And none of it adapts. The schedule is a plan written before the first step, indifferent to what the loss then does, kept because it is predictable rather than because it is right.\nThe figures in this article follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/articles/learning-rate-schedules/","summary":"\u003cp\u003eThe learning rate is usually introduced as a number: one small constant, multiplied into every gradient and then left alone. Almost no large model has ever been trained that way. In practice it is a function of the step count — near zero for the opening few thousand steps, rising to a peak, then falling for the rest of the run — and the shape of that function is treated as somewhere between a solved problem and a superstition. It is neither. The opening ramp exists for a specific and slightly embarrassing reason to do with the optimizer\u0026rsquo;s own bookkeeping, and the conventional curve imposes a constraint so awkward that an entire alternative arrangement was invented to escape it. This piece is about why the rate has to move at all, what the two halves of the schedule are separately doing, and what the standard shape quietly costs.\u003c/p\u003e","title":"Learning-Rate Schedules — Deciding When to Slow Down"},{"content":"The quadratic cost of attention has attracted more attempts at circumvention than any other single fact in this field, and most of them have failed. The idea is a piece of algebra rather than an approximation heuristic: attention involves multiplying three matrices, matrix multiplication is associative, and doing the multiplications in the other order turns a quadratic cost into a linear one. That much is exactly true and costs nothing. What costs something is the softmax sitting in the middle, which blocks the reassociation and must be removed or replaced — and the entire history of this line of work is the history of discovering what the softmax was doing that nobody had noticed. This piece is about the family that has come closest, about what the softmax turned out to be doing, and about why \u0026ldquo;closest\u0026rdquo; has been such a moving target.\nThe reassociation Set aside the softmax for a moment and write attention as three matrices multiplied together. Queries $Q$ and keys $K$ are $n \\times d$, where $n$ is the sequence length and $d$ the head width, and values $V$ is $n \\times d$. The computation is $(QK^\\top)V$.\nDo it in that order and the intermediate object $QK^\\top$ is $n \\times n$ — the score matrix, quadratic in sequence length, the source of all the trouble. The cost is $O(n^2 d)$.\nBut matrix multiplication is associative, so $(QK^\\top)V = Q(K^\\top V)$. Do it the second way and the intermediate object $K^\\top V$ is $d \\times d$ — a square matrix the size of the head width, with no dependence on sequence length at all. The cost is $O(n d^2)$.\nFor $n$ far larger than $d$, which is the case that matters, this is an enormous saving, and it is not an approximation. It is the same number computed in a different order.\nThe obstacle is that real attention is not $(QK^\\top)V$. It is $\\text{softmax}(QK^\\top)V$, and the softmax applies a nonlinearity — exponentiate, then normalize across each row — to the intermediate matrix. You cannot reassociate around a function applied to the thing in the middle. To get the saving, the softmax must go.\nFigure 1. Two orders of multiplication, two intermediate objects.\nReplacing the softmax with a feature map The standard move is to replace the exponentiated dot product with an inner product of transformed vectors. Where softmax attention weights position $j$ by $e^{q_i \\cdot k_j}$, linear attention weights it by $\\phi(q_i) \\cdot \\phi(k_j)$ for some function $\\phi$ applied to each vector separately.\nBecause $\\phi$ acts on the queries and keys individually rather than on their product, the reassociation is restored. Written out with the normalization included:\n$$y_i = \\frac{\\phi(q_i)^\\top \\sum_{j \\le i} \\phi(k_j) v_j^\\top}{\\phi(q_i)^\\top \\sum_{j \\le i} \\phi(k_j)}$$The sums run over all previous positions and do not depend on $i$ except through the range, which means they can be maintained incrementally. Keep a running matrix $S = \\sum \\phi(k_j) v_j^\\top$ of size $d \\times d$ and a running vector $z = \\sum \\phi(k_j)$, update both with each new token, and read off the output by multiplying. Generation costs a constant amount per token, and there is no KV cache that grows.\nWhich reveals what this family really is. That running matrix is a fixed-size state, updated additively as tokens arrive. Linear attention is a recurrent model wearing attention\u0026rsquo;s notation — the same object the article on state space models describes, reached from the opposite direction. One line of work started from recurrence and made it parallelizable; the other started from attention and made it recurrent; they met.\nThe choice of $\\phi$ has varied. A simple positive function such as an exponential-linear unit plus one, which guarantees the denominator stays positive. A random feature map chosen so the inner product approximates the exponential in expectation, which makes the method an unbiased estimator of true attention. A polynomial expansion. None has proved decisive, which is itself informative.\nFigure 2. The running state that makes it a recurrent model.\nWhat the softmax was doing The early versions of this were substantially worse than attention, and the diagnosis took a while. Three things turned out to have been lost.\nSharpness. The exponential in a softmax is what lets attention concentrate almost all its weight on one position. An inner product of non-negative features produces a much flatter distribution — every position with any feature overlap gets some weight, and the winner cannot pull far ahead. Attention\u0026rsquo;s ability to retrieve one specific thing depends on that sharpness, and linear attention blurs it.\nCapacity. The state is a $d \\times d$ matrix, and every token writes into it additively, forever, with nothing removed. After a few thousand tokens the accumulated sum is a superposition of thousands of outer products in a space that cannot keep them separable. This is the superposition argument applied unfavourably: the state has a finite capacity and the writes never stop, so old content is progressively overwritten by interference rather than by any decision.\nSelectivity. Nothing in the update can decide that a token is not worth remembering, or that an old memory should be released. Every token contributes equally and permanently.\nThe fixes that followed address these in order, and they are what made the family competitive. Adding a decay factor, so that the state is multiplied by something less than one at each step, gives forgetting — which frees capacity and matters more than its simplicity suggests. Making that decay depend on the input gives selectivity, the same ingredient that made state space models work. And treating the state as something to be written rather than accumulated — where each token performs a small corrective update that removes what was previously stored for a similar key before adding the new association — gives something closer to an actual memory with overwriting, which is the most recent and most promising direction.\nEach of these makes the model less like a reassociated attention and more like a learned recurrent memory. The algebra was the entry point, not the destination.\nFigure 3. What the softmax was doing, and what replaced it.\nWhy the field kept not adopting it For several years these methods were reported as matching attention while being far cheaper, and were not adopted. Three reasons, all instructive.\nThe first is that they were being compared against a badly implemented baseline. FlashAttention made exact attention several times faster and linear in memory, which moved the crossover point where a linear method becomes worthwhile out to sequence lengths beyond what most people use. An approximation must beat the best implementation of the thing it approximates, and for a while it was not being asked to.\nThe second is that the losses were real but concentrated. Average language modelling loss looked fine; retrieval, copying, and in-context learning from many examples did not. A headline metric can be matched while a specific and important capability is not, which is the evaluation problem in miniature.\nThe third is that a $d \\times d$ state per head sounds small and is not free — it must be held in fast memory during the scan, and naive implementations spent so much time moving it that the theoretical advantage evaporated. The gains only materialized once someone wrote the kernels with the same hardware-awareness that FlashAttention brought to exact attention.\nWhere it has landed is the same place the state space work landed, which is unsurprising given that the two turned out to be the same thing. Pure linear attention is not used for general-purpose models. Hybrid stacks — mostly linear or recurrent layers with a minority of exact attention layers — are used, increasingly in production, because they give constant-cost processing for most of the model while keeping a few places where the model can look something up exactly. The quadratic term was never the enemy; paying it at every one of eighty layers was.\nWhat to carry forward Linear attention begins from an exact observation: attention multiplies three matrices, matrix multiplication is associative, and computing $Q(K^\\top V)$ instead of $(QK^\\top)V$ replaces an $n \\times n$ intermediate with a $d \\times d$ one, turning quadratic cost into linear. The obstacle is the softmax applied to the middle term, which blocks the reassociation, so it is replaced by an inner product of separately-transformed queries and keys. The consequence is that the model maintains a fixed-size running state updated additively per token — which means linear attention is a recurrent model in attention\u0026rsquo;s notation, the same object state space models arrive at from the other direction. Removing the softmax loses three things: sharpness, since the exponential is what lets attention concentrate nearly all weight on a single position, and an inner product of non-negative features cannot; capacity, since a fixed state written to additively and never cleared accumulates interference until old content is unrecoverable; and selectivity, since no token can be judged not worth keeping. The repairs — a decay factor giving forgetting, input-dependent decay giving selectivity, and update rules that overwrite an existing association rather than accumulate on top of it — each move the method further from reassociated attention and closer to a learned memory. Adoption lagged because the comparison was against an unoptimized baseline until FlashAttention moved the crossover point, because the losses were concentrated in retrieval rather than visible in average loss, and because the state has to stay in fast memory to deliver its advantage. The outcome matches the state space story: not replacement, but hybrid stacks where most layers are constant-cost and a minority retain exact lookup.\nThe figures in this article follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/articles/linear-attention/","summary":"\u003cp\u003eThe quadratic cost of \u003ca href=\"/articles/attention/\"\u003eattention\u003c/a\u003e\n has attracted more attempts at circumvention than any other single fact in this field, and most of them have failed. The idea is a piece of algebra rather than an approximation heuristic: attention involves multiplying three matrices, matrix multiplication is associative, and doing the multiplications in the other order turns a quadratic cost into a linear one. That much is exactly true and costs nothing. What costs something is the \u003ca href=\"/articles/softmax/\"\u003esoftmax\u003c/a\u003e\n sitting in the middle, which blocks the reassociation and must be removed or replaced — and the entire history of this line of work is the history of discovering what the softmax was doing that nobody had noticed. This piece is about the family that has come closest, about what the softmax turned out to be doing, and about why \u0026ldquo;closest\u0026rdquo; has been such a moving target.\u003c/p\u003e","title":"Linear Attention — Trading Exactness for a Cost That Does Not Explode"},{"content":"The shape of a forward pass is easy enough to state. A token comes in, the tower of blocks works on it, and out the far end comes a single arrow — a few thousand numbers holding everything the model has assembled about what should happen next. But a vector is not a word, and somewhere between that arrow and the token that finally appears on your screen there is a step that is easy to wave past. That step produces one number for every entry in the vocabulary, and those numbers are the logits. They are the last thing the model computes for itself, before any of it is turned into something a person would recognize as a prediction. This piece is about where they come from, what their name means, and the one property of them that trips up almost everyone who meets them.\nThe problem with asking a network for odds directly It would be tidier if the model simply produced probabilities. Why not have the last layer output, for each of a hundred thousand vocabulary entries, the chance that it comes next, and be done with it?\nBecause a layer cannot make the necessary promises. What a layer does is multiply by a matrix and add a bias, and the result of that is an ordinary real number — it may be negative, it may be enormous, and there is no arrangement of weights that guarantees otherwise for every possible input. Probabilities have two obligations that are hard constraints rather than tendencies: none of them may be below zero, and all of them together must total exactly one. A hundred thousand numbers produced independently by matrix multiplication will satisfy neither. You could try to force it — squash each output into the range from zero to one and then divide by the total — but the squashing saturates, and a model whose scores have all been flattened against a ceiling has no way to express that one candidate is not merely likely but overwhelming.\nThere is a second reason, less obvious and more important. Think about what the tower actually does on the way up. Each block reads the running vector, decides something, and adds its contribution back. Evidence accumulates by addition: one attention head notices the sentence is about cooking and pushes a little toward food words; a later feed-forward layer recognizes a particular idiom and pushes harder toward one of them. That is the natural arithmetic of the architecture. But evidence does not combine additively on the scale of probabilities. Two independent pieces of evidence, each doubling the odds of something, multiply — and multiplication becomes addition only after you take a logarithm.\nSo the sensible thing is to let the network work in whatever units make its own arithmetic natural, let those numbers run unbounded in both directions, and convert to probabilities in one final step at the very end. The unbounded scores are the logits. As we are about to see, they turn out to live on precisely the logarithmic scale that makes the network\u0026rsquo;s additions correct.\nWhere the numbers come from: a shadow cast along a direction The conversion from arrow to scores is a single matrix, usually called the unembedding or the language-model head, and it is worth looking at one row at a time rather than all at once.\nEach row of that matrix is a vector the same size as the model\u0026rsquo;s internal arrow, and there is one row per vocabulary entry. So every word the model knows owns a direction in the same space the model has been working in all along. The logit for that word is the dot product of the final arrow with that word\u0026rsquo;s direction — which, geometrically, is the length of the shadow the arrow casts along it.\nThat single picture explains most of what follows. If the tower has assembled an arrow pointing strongly the way that \u0026quot; cheese\u0026quot; points, the shadow along \u0026quot; cheese\u0026quot; is long and its score is high. If the arrow is roughly perpendicular to \u0026quot; cheese\u0026quot; — the model has assembled something with nothing to do with cheese — the shadow is near zero. If the arrow points away, the score is negative. And because words that mean similar things sit in similar directions, an arrow that scores well for \u0026quot; cheese\u0026quot; will usually score decently for \u0026quot; butter\u0026quot; without anything in the model having arranged that on purpose. The scores of related words rise and fall together as a consequence of the geometry, not as a separate mechanism.\nNotice that this is the same trick as embedding , run backwards. On the way in, a token was converted into a vector; on the way out, a vector is compared against every token\u0026rsquo;s vector to see which it most resembles. Many models make the symmetry literal by weight tying — using one matrix for both, with the input embedding and the output direction for a given token being the same set of numbers. This saves an enormous quantity of parameters and generally costs nothing in quality, though the largest recent models often keep the two separate, having decided they can afford the memory and would rather let the two roles specialize.\nEnormous is the right word, and it is worth pausing on the size. The unembedding has one row per vocabulary entry and one column per dimension of the arrow: with a vocabulary of a hundred and twenty thousand and an arrow of four thousand numbers, that is nearly half a billion parameters in a single matrix, frequently the largest in the entire model. Computing the logits means half a billion multiplications for every token generated, and the resulting score vector — a hundred and twenty thousand numbers — is itself larger than any internal activation in the network. This is why training implementations go to some lengths to avoid ever holding the full logit tensor for a long sequence in memory at once, and why the final layer is a real engineering concern rather than an afterthought.\nFigure 1. A logit is a shadow cast along a word\u0026rsquo;s direction.\nA pinch of math: from scores to odds, and back Write $h$ for the final arrow the tower produces and $w_i$ for the direction belonging to vocabulary entry $i$. The logit for that entry is the plainest thing here:\n$$z_i = h \\cdot w_i$$A single dot product, repeated once for every word the model knows. To turn the resulting scores into odds, exponentiate each and divide by the total:\n$$p_i = \\frac{e^{z_i}}{\\sum_j e^{z_j}}$$Exponentiating does two jobs at once. It forces every number positive, disposing of the awkwardness that a score of minus four cannot be a probability, and it magnifies gaps, so that a score two points above its neighbour ends up not twice as likely but about seven times as likely.\nNow run the argument backwards, which is where the name comes from. Take two entries and ask about the ratio of their probabilities. The denominators are identical and cancel, leaving $p_i / p_j = e^{z_i - z_j}$, and taking a logarithm gives:\n$$\\log \\frac{p_i}{p_j} = z_i - z_j$$This is worth reading twice. The difference between two logits is exactly the log of the odds between the two outcomes. A logit is a log-odds — that is what the word means, borrowed from statistics, where the logit function $\\log\\!\\big(p/(1-p)\\big)$ has converted probabilities into unbounded scores since long before anyone stacked a transformer. It also settles the question left open in the previous section: the logarithmic scale on which the network\u0026rsquo;s additions are the right operation is the scale the logits already live on. When a layer adds two points to a token\u0026rsquo;s score, it has multiplied that token\u0026rsquo;s odds by about seven, and it has done so by simple addition, which is the only thing the architecture is good at.\nAnd notice what the identity does not contain. Add the same constant to every logit and each $z_i - z_j$ is unchanged, so every probability is unchanged; in the softmax, the constant factors out of numerator and denominator and cancels. The model\u0026rsquo;s output is completely blind to where the scores sit on the number line. You do not need to keep the algebra — the intuition is that logits are a scale of evidence with no fixed zero, like altitude measured with no agreed sea level.\nOnly differences are real That last observation deserves its own section, because it is the source of nearly every confusion about logits and also the reason they are useful.\nA single logit, quoted on its own, means nothing at all. If someone tells you the model gave \u0026quot; cheese\u0026quot; a logit of 14.2, the correct response is that you cannot do anything with that. Fourteen point two is high if the runner-up was at 9, and it is nothing at all if forty other tokens are at 15. The scale has no zero, no units, and no fixed range; it drifts between models, between layers, and between one position in a sentence and the next. Comparing a logit from one context against a logit from another is meaningless in the same way that comparing two altitudes measured from two different arbitrary baselines is meaningless.\nDifferences, on the other hand, are completely well defined, and they come with a natural interpretation. Since a difference is a log-odds ratio, one unit of logit difference is a factor of $e$ — a shade under three — in the relative odds of two tokens. Two units is about seven times. Five units is about a hundred and fifty. This gives a serviceable mental conversion: a gap of a fraction of a point between two candidates means the model is close to indifferent, a gap of three or four means it has largely made up its mind, and a gap of ten means the runner-up is not in the running.\nThis is why logit difference is the standard measuring instrument in interpretability work, and why it turns up as the metric of choice in ablation experiments. If you want to know whether a particular attention head is responsible for the model preferring one name over another, you take the difference between those two tokens\u0026rsquo; logits, switch the head off, and see how much the difference shrinks. Using the difference rather than either raw score removes everything that is not about the comparison you care about — including all the machinery that merely makes the model confident in general, and including the arbitrary offset. Using the probability instead would fold in the influence of a hundred thousand irrelevant tokens through the denominator, and would compress exactly the region you most want resolution in, since the difference between 99.0 and 99.9 percent is one full unit of logit and looks like almost nothing on a probability scale.\nFigure 2. Only differences are real — logit gaps as odds ratios.\nThe dials that act on the scores Everything that shapes how a model chooses its next word acts on the logits, in the window between the dot product and the softmax. This is the one place in the whole system where a person can reach in and intervene with a single addition or division.\nTemperature divides every logit by a constant before the exponential. Divide by something less than one and the gaps widen, the leader pulls away, and the model becomes decisive to the point of monotony; divide by something greater than one and the gaps close, the field bunches up, and unlikely words start winning. It is the same dial, doing the same thing, as the temperature used in distillation to soften a teacher\u0026rsquo;s output so a student can see the shape of its uncertainty — there it is applied to make the fine print visible, here to make the model more or less adventurous, but the arithmetic is identical.\nTruncation acts on the ranking rather than the spacing. Keeping only the highest $k$ scores, or only as many of the top scores as are needed to account for some share of the total probability, and setting the rest aside before sampling, is a way of allowing variety without allowing absurdity — the tail of a hundred thousand tokens holds a great deal of accumulated probability mass made up entirely of words that would be ridiculous.\nThen there is direct interference. Adding a fixed amount to a particular token\u0026rsquo;s logit makes it correspondingly more likely, and subtracting makes it less; this is what a logit bias setting in an interface does. The limiting case is the useful one: set a token\u0026rsquo;s logit to negative infinity and it becomes strictly impossible, since its exponential is exactly zero. Doing that to every token that would violate a grammar is how constrained decoding works — how a model can be made to emit only valid JSON, or only one of five permitted answers, with a guarantee rather than a request. The model is not being persuaded to comply. The alternatives have been removed from the ballot.\nReading the scores early: the logit lens Here is a trick that is genuinely lovely, and it follows from nothing more than the observation that the unembedding is just a matrix you can apply whenever you like.\nThe tower does not change the shape of the arrow. The vector flowing through it after block four has exactly the same dimensions as the vector that comes out of block thirty-two. So there is nothing stopping you from taking the half-finished arrow at some intermediate layer and multiplying it by the unembedding anyway, to read off the logits the model would have produced had it stopped there. This is the logit lens, and what it shows is that the model\u0026rsquo;s prediction is not computed at the end but drafted early and revised continuously.\nRun it layer by layer on a sentence and you can watch the answer come into focus. In the early blocks the top candidates are generic and often just reflect the current token. Somewhere in the middle the right kind of word starts to surface — the right part of speech, the right semantic neighbourhood — and in the upper blocks a specific token climbs and the field settles. Occasionally you catch a reversal, where a plausible early leader is actively suppressed by a later layer that has noticed something. It makes the residual stream legible as what it is: a running draft in the model\u0026rsquo;s own vocabulary, which every block reads, amends, and passes on.\nThe technique comes with caveats, and it is honest to name them. Intermediate arrows have not been through the final normalization the unembedding expects, so the readings are distorted, and various refinements exist that learn a small correction for each layer before applying the lens. Some layers are simply not writing in a form the unembedding can read, and the output is noise. But as a way of seeing that depth is refinement rather than a single enormous computation at the end, it is hard to beat.\nFigure 3. The logit lens — a draft written early and revised.\nTwo practical notes Exponentiating a logit of, say, ninety produces a number too large for the arithmetic to hold, and the result is an infinity that poisons everything downstream. The universal fix exploits the shift invariance we established above: subtract the largest logit from all of them before exponentiating. Every score becomes zero or negative, every exponential lands safely between zero and one, and because the shift cancels, the answer is exactly unchanged. Every softmax implementation in every framework does this silently.\nRelatedly, the training loss is always computed directly from the logits and never from probabilities that were computed first. Taking the logarithm of a small probability that has already been rounded to the nearest representable number loses precision that cannot be recovered, whereas the combined operation of softmax-then-logarithm simplifies algebraically into something stable. If you have ever wondered why the loss function in a codebase is named for cross-entropy with logits, this is why: it is not being handed probabilities and it does not want them.\nWhat to carry forward The logits are the model\u0026rsquo;s raw scores — one per vocabulary entry, produced by taking the final arrow out of the tower and dotting it against a direction belonging to each token, so that a logit is the length of the shadow the model\u0026rsquo;s assembled state casts along the word in question. They are unbounded because the network\u0026rsquo;s arithmetic is additive and cannot promise otherwise, and the softmax converts them into odds at the very end. The name is exact rather than decorative: the difference between two logits is the logarithm of the odds between those two outcomes, which means logits are the scale on which the tower\u0026rsquo;s layer-by-layer additions correspond to multiplying odds. The single most important consequence is that only differences carry meaning. Adding a constant to every logit changes no probability at all, so a lone score quoted in isolation is uninterpretable, while a gap of one unit is reliably a factor of about three in relative odds — which is why logit difference is the standard yardstick in interpretability work and the metric of choice for ablation. Every knob that shapes generation acts here, in the gap before the softmax: temperature divides the scores, top-$k$ and top-$p$ truncate them, a bias adds to one of them, and negative infinity removes a token from consideration outright, which is how constrained decoding enforces a format rather than requesting one. And because the unembedding is only a matrix, it can be applied to a half-finished arrow partway up the tower — the logit lens — revealing the prediction as a draft that is written early and revised block by block, rather than computed in one stroke at the end.\nThe figures in this article follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/articles/logits/","summary":"\u003cp\u003eThe shape of a forward pass is easy enough to state. A token comes in, the tower of blocks works on it, and out the far end comes a single arrow — a few thousand numbers holding everything the model has assembled about what should happen next. But a vector is not a word, and somewhere between that arrow and the token that finally appears on your screen there is a step that is easy to wave past. That step produces one number for every entry in the vocabulary, and those numbers are the \u003cem\u003elogits\u003c/em\u003e. They are the last thing the model computes for itself, before any of it is turned into something a person would recognize as a prediction. This piece is about where they come from, what their name means, and the one property of them that trips up almost everyone who meets them.\u003c/p\u003e","title":"Logits — The Scores Beneath the Odds"},{"content":"A model advertised as handling a million tokens is making two claims, and only one of them is usually true. The first is that it will accept an input that long without failing — a claim about position encodings, memory, and serving infrastructure, and one that is now routinely satisfied. The second is that it will use that input, attending to the relevant part of it as reliably at position nine hundred thousand as at position nine hundred. That claim is much weaker than the marketing implies, and the gap between the two is where most disappointment with long-context models lives. This piece is about what it takes to extend a window, what breaks when you do, and how to tell the difference between a model that can hold a document and one that can read it.\nThree separate walls It helps to separate the obstacles, because they were solved at different times by different means and are frequently conflated.\nThe memory wall was the first. Materializing the attention score matrix costs memory proportional to the square of the length, which put a hard ceiling on what could be run at any price. FlashAttention removed it by never writing the matrix down, dropping attention\u0026rsquo;s memory cost to linear. Note what this did not do: the time remains quadratic, so long contexts became possible without becoming cheap.\nThe position wall is that a model trained on four thousand tokens has never seen position four thousand and one. With learned position embeddings there is simply no entry in the table. With rotary embeddings there is a well-defined rotation, but it is one the model has never encountered, and performance degrades sharply past the training length. This is the wall that extension techniques address.\nThe serving wall is the KV cache , which grows linearly with length and is per-conversation rather than shared. This is what makes long contexts expensive in production rather than merely slow, and it is unaffected by anything the other two solutions do.\nOnly the second is really about the model. The other two are engineering, and they are the ones that got solved most convincingly.\nFigure 1. Three walls, three different fates.\nExtending the window The standard route is to train on short sequences, which is cheap, and then extend, which is comparatively cheap too. Nobody pretrains at a million tokens; the quadratic cost makes it absurd.\nThe mechanism exploits a property of rotary embeddings. Position enters as a rotation whose angle is proportional to position, so extending the range means presenting the model with angles beyond anything it saw. Position interpolation rescales instead: divide every position index by some factor, so that a context four times longer maps onto the same range of angles the model was trained on. Positions that were one apart are now a quarter apart, which the model handles gracefully, having seen fractional-seeming relationships throughout training. The article on rotary embeddings describes this as re-spacing the seats so an oversized room looks familiar.\nPlain interpolation has a flaw, and fixing it is where the refinements come in. RoPE uses many frequencies simultaneously — fast-rotating dimensions that distinguish nearby tokens, slow ones that encode long-range position. Interpolating all of them equally crushes the fast ones, degrading the model\u0026rsquo;s ability to tell adjacent tokens apart, which is a capability it uses constantly. The better schemes therefore rescale selectively: leave the high-frequency dimensions alone, since local relationships have not changed and those dimensions are not running out of range, and stretch only the low-frequency ones that actually need to cover new ground. Adjusting the base frequency of the rotation accomplishes much the same thing, and the two views turn out to be closely related.\nWith a good scheme, a few billion tokens of continued training on long documents — a fraction of a percent of pretraining — extends a model by ten or thirty times. This is why long-context versions of models appear so quickly after the originals. It is a modest post-hoc procedure, not a different model.\nAttention dilutes Here is the part that the extension techniques do not fix, and it follows from the softmax .\nAttention distributes a fixed total of one across every position available. With a hundred candidates, an irrelevant one might receive a hundredth of the weight; with a hundred thousand, the same irrelevant position receives a hundred-thousandth, but there are now a thousand times as many of them. The aggregate weight sitting on material that does not matter grows with the length of the context, and the signal from the one passage that does matter has to compete against an ever-larger sum of noise.\nThis is not a bug in an implementation. It is what normalizing over more candidates does, and it means that a model\u0026rsquo;s ability to pick out a relevant fact from a haystack degrades as the haystack grows even when nothing else has changed. The effect compounds with a second one: models attend more reliably to the beginning and the end of their input than to the middle, a pattern robust enough to have its own name — lost in the middle. The beginning benefits from the attention-sink machinery and from being visible to every position; the end benefits from recency and from the local-attention heads that dominate the lower layers. Material placed halfway through a long document is attended to least, which has the entirely practical consequence that where you put something in a prompt affects whether the model uses it.\nFigure 2. Dilution and the lost middle.\nMeasuring it honestly The standard test is to hide a sentence in a long document and ask a question only that sentence answers. Models pass this at impressive lengths, and passing it means less than it appears.\nThe task is retrieval of a lexically distinctive string, and it is close to the easiest thing a long context can be asked to do. The planted sentence is usually semantically out of place, which makes it findable by surface features; only one fact must be located; and nothing has to be combined. A model can pass at a million tokens while being unable to answer a question requiring two facts from opposite ends of the same document.\nThe tests that discriminate ask for more. Multiple needles that must all be found and related. Aggregation across the whole input rather than a lookup — counting occurrences, tracking a variable through a long program, following a character\u0026rsquo;s state through a novel. Questions whose answers depend on material that is not lexically distinctive. Performance on these degrades much earlier than the single-needle result suggests, often falling off well before the advertised limit and sometimes before a quarter of it. A useful rule when evaluating a claim about context length is to ask what task was used to support it, because the answer is usually the easy one.\nThere is also a plain economic point that gets lost. Even where a long context works, it is not free: the cache cost scales with length, so a conversation carrying two hundred thousand tokens of history occupies memory that could have served many shorter ones. Filling a context because it is available is a real expense.\nFigure 3. What the needle test hides.\nArchitectural alternatives Two directions attack the cost rather than working around it.\nSparse and windowed attention restricts which positions may attend to which. Most heads see only a local window; a few keep global reach. Since a great deal of language is local, this loses less than one might expect, and interleaving local and global layers in a fixed ratio has become common in production models. It makes cost linear in length for most of the layers, and it is a genuine approximation — information more than a window away can only travel by hopping through intermediate positions across layers.\nThe more radical option is to abandon the growing cache entirely and carry a fixed-size summary of the past instead, which is what state space models do, at the cost of no longer being able to look anything up exactly. Hybrid designs that interleave both kinds of layer are currently the most credible attempt to get long-range capability without a cache that grows forever.\nAnd there is the option of not extending the context at all. Retrieval puts a search step in front of the model and shows it only the relevant few thousand tokens, which is cheaper than a long context, easier to update, and easier to attribute — at the cost of depending on the retriever having found the right material. The two are often posed as competitors and are better read as answers to different questions: retrieval when the corpus is large and the query is focused, long context when the material is genuinely one document that must be considered as a whole.\nWhat to carry forward A long-context claim conflates two things: whether a model accepts the input, and whether it uses it. Three separate walls stood in the way, and they had different fates — the quadratic memory of the attention matrix, removed by never materializing it, though the time remains quadratic; the position encodings, which have no meaning past the training length; and the cache, which grows linearly per conversation and is what makes long contexts expensive to serve rather than merely slow. Only the second is really about the model, and it is addressed by interpolating positions so a longer sequence maps onto the range of rotations the model was trained on — selectively, leaving the high-frequency dimensions that distinguish nearby tokens alone and stretching only the slow ones — after which a small amount of continued training extends a window tenfold or more, which is why long-context variants appear so quickly. What extension does not fix is dilution: attention normalizes over every available position, so the aggregate weight on irrelevant material grows with the context, and models additionally attend less reliably to the middle of their input than to either end. Benchmarks obscure this, because locating one lexically distinctive planted sentence is nearly the easiest long-context task there is; tests requiring several facts to be combined, or aggregation over the whole input, degrade far earlier than the advertised limit. The alternatives are to restrict attention to local windows with a few global layers, to replace the growing cache with a fixed-size summary, or to retrieve the relevant few thousand tokens and skip the long context entirely — which is a different answer to a different question rather than a competitor.\nThe figures in this article follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/articles/long-context/","summary":"\u003cp\u003eA model advertised as handling a million tokens is making two claims, and only one of them is usually true. The first is that it will \u003cem\u003eaccept\u003c/em\u003e an input that long without failing — a claim about position encodings, memory, and serving infrastructure, and one that is now routinely satisfied. The second is that it will \u003cem\u003euse\u003c/em\u003e that input, attending to the relevant part of it as reliably at position nine hundred thousand as at position nine hundred. That claim is much weaker than the marketing implies, and the gap between the two is where most disappointment with long-context models lives. This piece is about what it takes to extend a window, what breaks when you do, and how to tell the difference between a model that can hold a document and one that can read it.\u003c/p\u003e","title":"Long Context — When the Window Outgrows the Attention"},{"content":"A pretrained model arrives as billions of knobs, nudged downhill across a vast landscape until they settle into a capable configuration. It is rarely the configuration anyone wants. A model may need to answer in a house style, to reason about legal contracts, to write to a particular register, or to cover a speciality the general model knows only vaguely. The obvious fix is to keep training it on more specific data — a step called fine-tuning — and fine-tuning works. The trouble is that fine-tuning a giant model the ordinary way means adjusting all of its billions of knobs, which demands the same crushing hardware, memory and expense as the original training run, and produces a second full copy of an enormous model for every task. LoRA — short for Low-Rank Adaptation — sidesteps both costs by training a small set of new knobs and leaving the original untouched. This piece is about how that small set can be enough, what it saves, and what it cannot change.\nThe problem with fine-tuning the whole thing Picture the situation concretely. A large model\u0026rsquo;s knowledge lives in great grids of numbers — the weight matrices that sit inside every block of the tower. Each grid can hold millions of numbers, and there are hundreds of them. Ordinary fine-tuning reaches into every one of those grids, adjusts every number, and saves the whole altered set as a new model. Ten specialized tasks, ten full copies. Each may run to hundreds of gigabytes. Every one of those runs also needed enough memory to hold not just the model but all the bookkeeping training demands — the momentum and per-knob step sizes the optimizer keeps — several times over.\nThis is wasteful, and there is a reason to suspect it is wasteful by a large factor. Adapting a model to a new task is not teaching it language from scratch; it already knows language. The adjustment is small — a nudge in a particular direction. The suspicion, which turns out to be correct, is that this adjustment is far simpler than the giant grids it is applied to. A simple change should be cheap to describe.\nThe key idea: the change is smaller than it looks Here is the observation at the heart of LoRA. When a weight grid is fine-tuned, the difference between the original grid and the fine-tuned one — call it $\\Delta W$, the change made — has a hidden simplicity. Though it is a huge grid of numbers, that grid of changes can be reconstructed almost perfectly from far fewer numbers than it appears to contain. In the language of linear algebra, $\\Delta W$ has low rank: it carries much less independent information than its size suggests.\nConsider a large printed grid in which every row is a multiple of one single pattern — the second row twice the first, the third five times the first, and so on. The grid looks as though it holds hundreds of independent values. It holds one pattern and a list of multipliers. Nothing else is stored. That is what low rank means: a big grid that is the product of two much narrower ones. LoRA\u0026rsquo;s bet is that the change a fine-tune wants to make has this character.\nBuilding the change from two narrow grids So instead of learning the full change $\\Delta W$ directly, LoRA learns two small grids whose product is the change. Suppose the original weight grid is $d$ rows by $d$ columns — a thousand by a thousand, a million numbers. LoRA introduces two new grids: a tall narrow one, $A$, of size $d \\times r$, and a short wide one, $B$, of size $r \\times d$, where $r$ — the rank — is a small number like 8 or 16. Their product $B A$ has the same shape as the original grid, $d \\times d$, so it can stand in for the change:\n$$W_{\\text{adapted}} = W_0 + \\Delta W = W_0 + B A$$Here $W_0$ is the original, frozen weight grid. It never changes. All the learning happens inside $A$ and $B$, and the arithmetic settles the case. The full grid holds $d \\times d$ numbers — a million if $d$ is a thousand. $A$ and $B$ together hold $2 \\times d \\times r$ — with $r = 8$, about sixteen thousand, roughly one-sixtieth. Across a whole model the trainable share is often well under one percent, and it recovers most of the benefit of a full fine-tune.\nFigure 1. A million numbers rebuilt from sixteen thousand.\nWhat actually happens during training The picture during fine-tuning is now simple. The original model is frozen, so none of its knobs move and none of the heavy training bookkeeping has to be stored for them. Alongside each weight grid sits an $A$ and $B$ pair, and only these are trained by the ordinary downhill walk. The trainable part is small, so the memory and compute demands collapse to what one modest machine can hold. The alternative was a cluster.\nOne detail carries more weight than its size suggests. At the start of training the adapted model should behave exactly like the original, so that adaptation begins from a clean slate. LoRA arranges this by initializing $B$ to all zeros, so $BA$ starts at zero and adds nothing. At the first step the model is untouched; from there $A$ and $B$ take up the adjustment the task calls for. A scaling factor, often written $\\alpha / r$, sets how strongly the learned change is applied.\nTwo payoffs: cheap to train, and cheap to keep The first payoff is the one the argument has been driving at: training becomes affordable, because the optimizer works on a small set of knobs instead of billions. The second is as valuable and more often overlooked. Since the original model is never modified, there is no new copy per task. What gets stored is the $A$ and $B$ grids — often a few megabytes — as an adapter attached to the shared, frozen base. One base model sits in memory while dozens of adapters take turns: one for legal writing, one for a house register, one for medical questions. Swapping tasks costs a few megabytes of transfer rather than a second model.\nThe adapter can also be merged permanently into the base: because $W_0 + BA$ is another grid of the same shape, that sum can be computed once and baked in, producing a fine-tuned model with no extra run-time cost. The two properties do not survive together. Merging is what buys the zero-cost forward pass, and it is also what discards the interchangeability the adapter existed for.\nFigure 2. A copy per task against one base and many adapters.\nWhy it works so well, and where it stops It would be reasonable to expect that training a thousandth of the knobs buys a thousandth of the quality. It does not. On a great many tasks LoRA comes close to full fine-tuning, and sometimes matches it. The reason returns to the opening observation: adapting a model that already knows language is a small, structured intervention rather than a rebuild. The base has done the work of learning how words relate, and the task-specific adjustment is simple enough to live in two narrow grids. The tool is matched to the size of the job. That is the whole of it.\nWhich is also where it stops. The rank $r$ is fixed before the run, and an adaptation whose true change is not low-rank will underfit however long it trains. The diagnosis is a sweep: raise $r$ and see whether quality follows. That costs the runs the method was meant to save.\nThe deeper point is that LoRA changes the price of fine-tuning and not what fine-tuning is. It is a parameterisation, not an objective, so it carries whatever ceiling the objective it serves already had — trained on demonstrations it inherits the ceiling the article on supervised fine-tuning describes, and that article is also clear that preference methods and verifiable rewards are what break it. LoRA can parameterise those too. What the frozen base does constrain is the kind of change on offer: a few thousand numbers per matrix reshape behaviour and format far more readily than they install knowledge the weights never held.\nFigure 3. The rank is fixed before the evidence arrives.\nWhat to carry forward LoRA adapts a giant model to a new task without touching the model itself. It freezes the original weights and learns only the change, exploiting the fact that this change has low rank — it can be rebuilt from two small grids, $A$ and $B$, whose product $BA$ stands in for a full weight adjustment, roughly one-sixtieth of the numbers at $r = 8$. Training only those two grids brings the memory and cost of fine-tuning down to a single machine, and because the base is never altered, each task is saved as a small attachable adapter rather than a duplicate model; merging trades that interchangeability back for a free forward pass. It is what turned fine-tuning from a capital expense into a routine one. The trade underneath is that the rank is fixed before any evidence arrives, and a change that is not low-rank plateaus quietly, diagnosable only by spending the runs the method saved. And the saving is in the price of the stage rather than its reach: a few thousand numbers per matrix move behaviour and format much more readily than knowledge, which is why QLoRA went after the size of the frozen base and not the ceiling above it.\nThe figures in this article follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/articles/lora/","summary":"\u003cp\u003eA pretrained model arrives as billions of knobs, nudged downhill across a vast landscape until they settle into a capable configuration. It is rarely the configuration anyone wants. A model may need to answer in a house style, to reason about legal contracts, to write to a particular register, or to cover a speciality the general model knows only vaguely. The obvious fix is to keep training it on more specific data — a step called \u003cem\u003efine-tuning\u003c/em\u003e — and fine-tuning works. The trouble is that fine-tuning a giant model the ordinary way means adjusting \u003cem\u003eall\u003c/em\u003e of its billions of knobs, which demands the same crushing hardware, memory and expense as the original training run, and produces a second full copy of an enormous model for every task. LoRA — short for \u003cem\u003eLow-Rank Adaptation\u003c/em\u003e — sidesteps both costs by training a small set of new knobs and leaving the original untouched. This piece is about how that small set can be enough, what it saves, and what it cannot change.\u003c/p\u003e","title":"LoRA — Teaching an Old Model New Tricks Without Retraining It"},{"content":"The loss curves that appear in technical reports are smooth. They descend, they flatten, and they suggest that training a large language model is a matter of starting the machinery and waiting. The curves kept by the people running those jobs look different. Somewhere around step forty thousand, with nothing changed in the configuration, the loss jumps — not drifts, jumps — by a whole nat or two inside a handful of steps, and then does one of three things: claws its way back over a few thousand steps, settles permanently above the trajectory it left, or keeps rising until the model emits one token forever. This is a loss spike, and it is why a frontier run is watched on a rota rather than left alone over a weekend. This piece is about what happens when a run falls over, why the proposed causes are still argued about, and what the available defences cost.\nWhat a spike looks like from the logs The defining feature is the timescale. A run whose loss has descended for tens of thousands of steps, smoothly enough to fit a scaling law through, moves one or two nats in five or ten steps. On a plot of the whole run it is a vertical line. Zoom in and the gradient norm has almost always gone first, one step at ten or a hundred times its recent value, with the loss following immediately, because the step that norm produced is already in the weights.\nWhat follows is not one behaviour but three. Sometimes it recovers unaided, descending more steeply and rejoining its old trajectory within a few thousand steps. Sometimes it recovers most of the way and stops, tracking parallel to where it was but permanently above it, and the run finishes worse than intended. And sometimes it does not recover: the loss climbs to what guessing would give, or ceases to be a number, and what remains puts all its mass on one token regardless of input.\nThe reason this is studied rather than shrugged at is arithmetic. A large pretraining run occupies thousands of accelerators for weeks, costs millions of dollars, and cannot be repeated with a different seed to see whether the trouble goes away. There is one run. If it spikes at the halfway mark and lands in the second outcome, the money has been spent and the model that emerges is not the one budgeted for.\nFigure 1. Three things a spike does next.\nSmall models are hard to destabilize and large ones are not The first thing any explanation has to account for is the size dependence, and it is stark. Small models are remarkably difficult to break. Set the learning rate too high and a model of a few hundred million parameters merely trains badly, converging to a worse loss without the characteristic discontinuity; feed it a corrupted batch and it absorbs it. The same configuration at ten times the parameters will spike, sometimes repeatedly, sometimes fatally.\nThis is what makes spikes a property of scale rather than a defect in anyone\u0026rsquo;s code. A bug would show at every size and be findable by ordinary means. Instead, instability appears gradually as width and depth grow, in a configuration validated cheaply and never doubted. The validation was honest; the transfer was the problem. Whatever the mechanism, its severity must grow with the model, which rules out a good many otherwise plausible stories.\nFour mechanisms wear the same symptom The specific culprit is still argued over, and groups running comparable models have reported different dominant causes. What is not argued over is the shape of the failure: something develops an unusually large magnitude, one step is taken far too big, and the resulting weights are either recoverable or not. Four candidates produce that shape, they are not exclusive, and the argument persists because a spike leaves too little evidence to say which it was.\nThe leading candidate is attention logit growth. Query-key dot products are constrained by nothing. Over a long run some heads acquire larger projection norms, the scores grow with them, and the softmax saturates — one weight goes to essentially one, the rest to essentially zero. A saturated softmax has almost no gradient, so the head stops receiving a corrective signal and the drift that caused the saturation goes unopposed. That is the self-reinforcing part. The output, meanwhile, has become a near one-hot selection of one position\u0026rsquo;s value vector, and selection is discontinuous: a small change in the projections moves which position wins, and the output jumps by the whole difference between two value vectors. Untrainable and violently sensitive at once. It worsens with width, since a dot product over more dimensions sums more terms and saturation arrives sooner.\nThe second is the optimizer\u0026rsquo;s own memory. An adaptive optimizer divides each step by a slow-moving estimate of that parameter\u0026rsquo;s recent gradient magnitude, slow by design so the sense of scale stays steady. That is the wrong property when a batch delivers a gradient a hundred times larger than anything recent: the denominator still reflects the old magnitudes, the numerator the new one, and the step is enormous. The estimate catches up within a few dozen steps, which is why the anomaly ends quickly in the log, but the damage was in the weights first. This is why practitioners shorten that memory for large models by lowering the second-moment decay constant. A shorter memory still has a memory. It is a mitigation, not a cure.\nThe third is numerical range. A sixteen-bit format with five exponent bits leaves little headroom above ordinary activation magnitudes, and an overflow anywhere produces an infinity that propagates through the backward pass into the gradient, into the optimizer state, and from there into every weight it touches. One bad value poisons the model in a single update. The shift to mixed-precision formats with the exponent range of single precision removed this class almost entirely and left the others as they were, which is the most useful evidence in the subject: several distinct mechanisms were plainly wearing one symptom.\nThe fourth, and the least satisfying, is the data. A single pathological batch can produce a gradient unlike anything the run has seen: ten thousand repetitions of one token, binary that survived the filters, a script the tokenizer handles badly. This is real, it is the easiest of the four to fix, and it explains fewer spikes than people hope. Careful data curation removes the worst offenders, and runs still spike on batches later inspected and found unremarkable.\nA pinch of math: what growing logits buy on both sides Write the attention weights over a row of scores $z_1 \\ldots z_n$ in the usual way,\n$$p_i = \\frac{e^{z_i}}{\\sum_j e^{z_j}}$$with the head\u0026rsquo;s output the weighted sum $o = \\sum_i p_i v_i$. Two derivatives matter, and they move in opposite directions as the scores grow.\nThe first is how much a weight responds to a score:\n$$\\frac{\\partial p_i}{\\partial z_i} = p_i(1 - p_i), \\qquad \\frac{\\partial p_i}{\\partial z_j} = -\\,p_i p_j \\quad (j \\neq i)$$Every one of those terms collapses toward zero once any single $p_i$ approaches one: the diagonal term is maximized at $p_i = 1/2$ and vanishes at both extremes, and the off-diagonal terms are products of a number near one with numbers near zero. A saturated softmax has almost no derivative anywhere, so whatever gradient arrives from above is multiplied by something near zero and never reaches the projections that produced the scores.\nThe second is how much the output responds to which score is largest. As $p$ becomes one-hot on index $a$, the output approaches $v_a$. Move the largest score from $a$ to $b$ and the output moves by $v_a - v_b$ — a whole difference between two value vectors, of order one, with no tendency to shrink as the logits grow, since nothing has trained those vectors to resemble each other.\nGrowing logits therefore buy a vanishing gradient and a rising discontinuity at once, in the same head. The smooth stand-in for the maximum has quietly become the maximum: flat nearly everywhere, with a cliff. That combination is the mechanism.\nFigure 2. A vanishing gradient and a rising cliff, together.\nEvery mitigation charges something for its help None of the defences in use is free, and the honest way to present them is with the bill attached.\nGradient clipping by global norm is nearly universal and almost free. Compute the norm of the whole gradient and, if it exceeds a threshold, rescale down to that threshold before the optimizer sees it, turning a catastrophic step into a merely large one. It also changes the problem being solved: on any step where it fires, the direction is preserved but the magnitude was chosen by an engineer, not by the loss surface.\nNormalizing the queries and keys before the dot product attacks the first mechanism directly, capping how large a score can become whatever the projections do. It costs two small normalization operations per head per token and removes a degree of freedom, since a head can no longer express confidence through score magnitude alone.\nAn auxiliary loss on the softmax normalizer — penalizing the logarithm of the denominator, so logits are pushed back whenever they drift upward — works, and adds a term to the objective that nobody wanted. Its coefficient is one more thing to tune, and it trades a little of the primary loss for stability.\nLowering the learning rate, or the second-moment constant, or both is the mid-run response of last resort, and it works by making everything slower. A schedule reduced for safety reaches a worse loss for the same compute, and since the counterfactual was never run, what was surrendered cannot be measured.\nThe brute-force operational answer is what large runs actually do. Checkpoint every few hundred steps; when the monitoring catches a spike, stop, rewind a few thousand steps, skip the batches that were in flight, and resume. It is unsatisfying, it costs storage and the discarded steps, and it works often enough to be standard procedure rather than a fallback.\nWhat is not available is prediction. No signal reliably says a spike is coming before the gradient norm moves, there is no agreement on which mechanism dominates at a given scale, and no theory says whether a recovered run ends where one that never spiked would have. Practice here is a long way ahead of understanding.\nFigure 3. Four mechanisms, one symptom, and what each defence costs.\nWhat to carry forward A large run can descend smoothly for tens of thousands of steps and then lose one or two nats in five, after which it either climbs back, settles permanently at a worse level, or degenerates into a model that emits one token forever. All three occur, and since the run cannot be repeated, the second outcome is as expensive as the third. This is a phenomenon of scale rather than a defect: small models resist destabilization, and a configuration validated at a smaller size will spike at ten times that size, so the mechanism must involve magnitudes that grow with the model. Four mechanisms fit, and they are not exclusive. Query-key dot products drift upward until the softmax saturates, at which point the head has almost no gradient while its output is a near one-hot selection that swings discontinuously. An adaptive optimizer divides its step by a slow estimate of gradient scale, so a gradient far above recent history produces a step far above any intended size, and the estimate corrects itself only after the damage is done. A narrow exponent range permits an overflow whose infinity propagates into every parameter; a wider-range format removed that class and left the rest untouched. A pathological batch can do it too, less often than hoped. The defences all charge something: clipping changes the problem being solved whenever it fires, normalizing queries and keys costs a degree of freedom, a penalty on the softmax normalizer adds a term nobody wanted, and a lowered learning rate buys stability with loss. What large runs rely on is frequent checkpointing and rewinding on a spike, which is not a theory of anything. Nothing predicts a spike before the gradient norm moves, and nothing says whether a recovered run ends where an untroubled one would have.\nThe figures in this article follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/articles/loss-spikes/","summary":"\u003cp\u003eThe loss curves that appear in technical reports are smooth. They descend, they flatten, and they suggest that training a large language model is a matter of starting the machinery and waiting. The curves kept by the people running those jobs look different. Somewhere around step forty thousand, with nothing changed in the configuration, the loss jumps — not drifts, jumps — by a whole nat or two inside a handful of steps, and then does one of three things: claws its way back over a few thousand steps, settles permanently above the trajectory it left, or keeps rising until the model emits one token forever. This is a \u003cem\u003eloss spike\u003c/em\u003e, and it is why a frontier run is watched on a rota rather than left alone over a weekend. This piece is about what happens when a run falls over, why the proposed causes are still argued about, and what the available defences cost.\u003c/p\u003e","title":"Loss Spikes — When a Training Run Falls Over"},{"content":"Memorization arrived in machine learning with a settled meaning and a verdict attached: a model that had fitted its training set too closely, which was bad and, better still, detectable — watch the held-out loss, and when it turns upward while the training loss keeps falling, stop. That diagnostic is still taught, still correct on its own terms, and almost beside the point for a model trained once over a corpus it will never exhaust. The held-out loss does not turn upward. Nothing on the dashboard looks wrong. The model has nonetheless retained enough of its training data to reproduce parts of it word for word, including passages that appeared a handful of times and strings nobody wanted kept. The old alarm stays quiet because it was wired for a different fault. This piece is about why the training objective asks for verbatim retention rather than merely permitting it, what determines which sequences receive it, and what the available remedies do and do not remove.\nNothing in the objective distinguishes a fact from a string Training a language model consists of raising the probability it assigns to text that actually occurred, position by position, measured by cross-entropy . Take the sentence the capital of France is Paris. Two different changes will reduce the loss on it. One is to acquire something general: a representation of countries and their capitals, wired into the syntax of that construction, which pays off again on Lisbon and Ottawa and every sentence shaped like this one. The other is to store the sequence — to arrange the weights so that this prefix makes this continuation likely, with no consequence anywhere else.\nBoth lower the loss. Gradient descent sees the loss and nothing else. It has no term expressing a preference for the general solution, no penalty on the specific one, no representation of the distinction at all; generalization happens only because the general solution is usually cheaper in parameters than storing every instance, and where it is not cheaper, storage wins. So memorization is not a defect in the pretraining objective . It is one of the things the objective asks for.\nMemorization is necessary, not merely tolerated. This cuts against the reflex, so it belongs first. A model must retain a great deal verbatim to be useful at all. Idioms are not derivable, names attach to people by convention, code syntax is a set of arbitrary strings that must be reproduced exactly or the program will not run, and quotations are quotations. Spelling is a fact about a word, and given how tokenization chops text into fragments, even ordinary orthography must be partly held rather than inferred. A model that memorized nothing would know nothing in particular. The question is never whether a model memorizes, but what, and which of it should not have been kept.\nFigure 1. Two ways to lower the loss on one sentence.\nWhat predicts whether a sequence is kept Retention is not uniform across a corpus, and the factors governing it can be ranked.\nDuplication dominates everything else. A sequence appearing many times in the training data is far more likely to be reproducible than one appearing once, and the relationship is steep enough that reducing duplicate counts moves reproducibility more than any other lever. It is why deduplication, as work on data mixtures reports, is the cheapest effective intervention. Every additional copy is another pass over the same tokens, applying the same pressure in the same direction.\nModel size pushes the wrong way. At equal exposure, a larger model retains more. Capacity is what storage consumes, so this is unsurprising, and it sits awkwardly beside the scaling results, because the extra parameters that buy better generalization also buy more literal retention. They are both what capacity is for.\nDistinctiveness is the uncomfortable factor. A bland sentence is protected by its own predictability: general competence already assigns it decent probability, so little residual loss is left for storage to claim. A random-looking string enjoys no such protection. No general pattern would produce it, so the only route to lowering its loss is to raise the probability of those tokens after that prefix, which is storage by definition. The sequences most exposed are therefore the ones that look like identifiers — keys, account numbers, unusual names, anything whose whole point is that it could not be guessed.\nRetention is measured by asking a trained model to continue text it was trained on. That is the whole mechanism of extraction as a research technique, and the detail worth giving here is the detail a security paper gives a vulnerability class: the mechanism, without the recipe. The shape of the findings is what matters. A non-trivial quantity of training data is recoverable from a deployed model, the quantity grows with model size, and the recoverable material skews toward duplicated or distinctive text. Extraction measures what is already present. It does not put it there.\nFigure 2. The three factors that decide what is kept.\nA pinch of math: why an incompressible string has to be stored The argument is short and it explains the distinctiveness effect. For a sequence of $k$ tokens, the loss contributed is\n$$L_{\\text{seq}} = -\\sum_{t=1}^{k} \\log p(x_t \\mid x_{","permalink":"https://www.fanfani.net/articles/memorization/","summary":"\u003cp\u003eMemorization arrived in machine learning with a settled meaning and a verdict attached: a model that had fitted its training set too closely, which was bad and, better still, detectable — watch the held-out loss, and when it turns upward while the training loss keeps falling, stop. That diagnostic is still taught, still correct on its own terms, and almost beside the point for a model trained once over a corpus it will never exhaust. The held-out loss does not turn upward. Nothing on the dashboard looks wrong. The model has nonetheless retained enough of its training data to reproduce parts of it word for word, including passages that appeared a handful of times and strings nobody wanted kept. The old alarm stays quiet because it was wired for a different fault. This piece is about why the training objective asks for verbatim retention rather than merely permitting it, what determines which sequences receive it, and what the available remedies do and do not remove.\u003c/p\u003e","title":"Memorization — What a Model Keeps Verbatim, and When That Matters"},{"content":"The article on QLoRA introduced the idea of describing a number in fewer bits, and applied it to a frozen model being adapted. Training a model from scratch in reduced precision is a harder problem, because a frozen weight only has to be read accurately while a weight under training has to accumulate millions of tiny corrections without any of them being lost to rounding. The tension is sharp: half-precision arithmetic runs several times faster and halves the memory traffic, which is the difference between a training run costing one month and three, but the updates the optimizer wants to apply are frequently smaller than the smallest change a half-precision number can represent. This piece is about how that tension is resolved, why the resolution hinges on a detail of how floating-point numbers are laid out, and what changed when the format itself was redesigned for the purpose.\nWhat a floating-point number spends its bits on A floating-point number divides its bits into two jobs. Some encode the exponent, which sets the scale — how large or small the number is, in powers of two. The rest encode the mantissa, the significant digits within that scale. Exponent bits buy range; mantissa bits buy precision.\nSingle precision, the traditional default, uses eight exponent bits and twenty-three mantissa bits in thirty-two, which is generous on both counts. The original half-precision format, fp16, uses five and ten in sixteen. Halving the total was the point, but notice which half was cut hardest: three exponent bits were surrendered, and the smallest positive number the format can represent normally is about $6 \\times 10^{-5}$, against roughly $10^{-38}$ for single precision. That is not a small reduction in range. It is thirty-three orders of magnitude.\nThis matters because gradients in a large network are small. Deep in a tower, for a parameter that barely affects the loss, a gradient of $10^{-8}$ is entirely ordinary. In fp16 that number is not small — it is zero. It underflows, silently, and the parameter receives no update at all. A training run can appear to be progressing while a substantial fraction of its gradients are being quietly discarded.\nFigure 1. Where three formats spend their bits.\nThree fixes, and why all three are needed The standard recipe for training in fp16 has three components, each addressing a different failure, and leaving any one out breaks it.\nA master copy in full precision. The optimizer keeps the authoritative weights in thirty-two bits and casts a half-precision copy for the forward and backward passes. The reason is accumulation. A weight of magnitude one, updated by $10^{-7}$, is unchanged in fp16 — the update is far below the gap between representable numbers at that scale, so adding it rounds straight back to where you started. Do that ten million times and the weight has still not moved, even though the corrections summed to something substantial. Keeping the master copy wide means every tiny update lands somewhere, and the accumulated total eventually becomes visible in the half-precision copy. This is why the memory arithmetic in the article on AdamW counts a four-byte master weight alongside the two-byte working weight.\nLoss scaling. To stop gradients underflowing, multiply the loss by a large constant before the backward pass. Every gradient is scaled up by that same constant — the chain rule guarantees it — lifting them out of the range where fp16 rounds to zero. Divide them back down before the optimizer applies them, and the mathematics is unchanged. The scale cannot be fixed in advance, because too small a value fails to rescue the smallest gradients and too large a value overflows the largest, so implementations adjust it dynamically: raise it steadily while things are fine, and on any step where a gradient comes back as infinity or not-a-number, discard that step entirely and halve the scale. A training run in fp16 therefore throws away an occasional batch by design, which is disconcerting to see in a log and is working as intended.\nSelective full precision. Some operations are performed in thirty-two bits regardless. Anything that sums many terms — the reductions inside normalization , the softmax denominator, the cross-entropy computation — accumulates rounding error in proportion to how many terms it adds, and in half precision the accumulated error can exceed the answer. The rule of thumb is that matrix multiplications are safe in low precision because the hardware accumulates them in a wider register internally, while explicit reductions in the model\u0026rsquo;s own code are not.\nThe format that made most of this unnecessary The three-part recipe worked and was universally used, and it was also fragile — loss scaling in particular is a moving part that can fail in ways that look like a bad learning rate. The eventual fix was to change the number format rather than to keep working around it.\nbfloat16 is also sixteen bits, but it allocates them differently: eight exponent bits and seven mantissa bits. It keeps single precision\u0026rsquo;s exponent field exactly and pays for it out of the mantissa. The result is a format with the same range as thirty-two-bit floating point and considerably less precision.\nThat is the correct trade for this problem, and understanding why is the point of this piece. Gradients do not need many significant digits. They are noisy estimates in the first place — the whole argument of the article on batch size is that the gradient is a sample, not a measurement — so knowing one to three decimal places rather than seven costs almost nothing. What gradients absolutely do need is to exist, and existing means being representable. Range is what they need; precision is what they can spare. The original format optimized the wrong one.\nWith bfloat16, gradients do not underflow, so loss scaling becomes unnecessary and disappears from the recipe entirely. The master copy remains, because the accumulation problem is about precision and bfloat16 has less of it than fp16 did. Selective full precision for reductions remains. But the fragile, dynamic, occasionally-run-destroying component is gone, and essentially all current large-model training uses bfloat16 for this reason.\nFigure 2. The fp16 recipe, and which parts bfloat16 retired.\nA pinch of math: why accumulation needs the wide copy The one part worth making precise is why a master copy is required even when range is not an issue, because it is the part people most often assume they can skip.\nA floating-point format with $m$ mantissa bits represents numbers near magnitude $x$ with a spacing of about $x \\cdot 2^{-m}$. Adding anything smaller than half that spacing leaves the value unchanged after rounding. So the condition for an update $\\Delta w$ to have any effect on a weight $w$ is:\n$$|\\Delta w| \\;\\gtrsim\\; |w| \\cdot 2^{-(m+1)}$$For bfloat16, $m = 7$, so an update must be roughly one part in two hundred and fifty of the weight\u0026rsquo;s own magnitude to register at all. A weight of magnitude $0.02$ therefore ignores any update below about $10^{-4}$ — and typical updates late in training are far smaller than that. In single precision, with $m = 23$, the threshold is one part in sixteen million, and the same updates land comfortably.\nThis is a different failure from underflow and is not fixed by having range. The number $10^{-7}$ is perfectly representable in bfloat16; it simply vanishes when added to something eight orders of magnitude larger. The master copy exists so that these vanishing updates accumulate somewhere they can be seen, and it is the reason low-precision training saves less memory than the bit count suggests — the weights are stored twice.\nFigure 3. The smallest update a weight will notice.\nEight bits, and where the line currently sits The obvious next step is eight bits, and it is now in production use on hardware that supports it, though with more care than sixteen required.\nThe formats offer a choice between range and precision — one with more exponent bits, one with more mantissa — and the practice is to use different formats in different places: the one favouring precision for weights and activations in the forward pass, the one favouring range for gradients in the backward pass, on the same reasoning that motivated bfloat16. Scaling becomes essential again, but per-tensor rather than global: each matrix carries its own scale factor, chosen from the recent history of its values, so that its numbers occupy the available range. Some designs go further and scale per block of a few dozen values, which handles the outlier dimensions described in the article on softmax — those enormous activations that support attention sinks would otherwise force a scale that crushes everything else in the tensor.\nWhat is not done in eight bits is the accumulation. Matrix products are computed from eight-bit inputs but summed in a wider register, the master weights stay in thirty-two bits, and the optimizer state stays wide. The pattern is the same one running through this whole subject and through QLoRA : store and multiply narrow, accumulate wide. Where the field will stop is not clear — four-bit training has been demonstrated and is not yet routine — but the principle has held at every step so far, which suggests that whatever precision the multiplications end up using, something behind them will be keeping a careful wider tally.\nWhat to carry forward Training in half the bits halves memory traffic and multiplies arithmetic throughput, but a weight being trained must absorb millions of corrections far smaller than itself, which is a harder demand than merely being read accurately. A floating-point format splits its bits between an exponent, which buys range, and a mantissa, which buys precision; the original half-precision format cut the exponent hard, so ordinary small gradients underflow silently to zero. The classical recipe patched this three ways: a full-precision master copy of the weights so tiny updates can accumulate somewhere rather than rounding away; loss scaling, multiplying the loss by a large dynamically-adjusted constant to lift gradients out of the underflow region, discarding any step that overflows; and keeping explicit reductions — normalization, softmax, the loss itself — in full precision, since summing many terms accumulates error in proportion to their number. bfloat16 made most of that unnecessary by reallocating the bits: full single-precision range, much less precision, which is the right trade because gradients are noisy estimates that need few significant digits but must at least be representable. Loss scaling disappeared with it; the master copy did not, because the accumulation problem is about precision, and an update smaller than roughly one part in $2^{m}$ of a weight\u0026rsquo;s magnitude vanishes on addition no matter how well the format can represent it in isolation. Eight-bit training now works using per-tensor or per-block scales and different formats for the forward and backward passes, and it obeys the same rule as everything before it: multiply narrow, accumulate wide.\nThe figures in this article follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/articles/mixed-precision/","summary":"\u003cp\u003eThe article on \u003ca href=\"/articles/qlora/\"\u003eQLoRA\u003c/a\u003e\n introduced the idea of describing a number in fewer bits, and applied it to a frozen model being adapted. Training a model from scratch in reduced precision is a harder problem, because a frozen weight only has to be \u003cem\u003eread\u003c/em\u003e accurately while a weight under training has to accumulate millions of tiny corrections without any of them being lost to rounding. The tension is sharp: half-precision arithmetic runs several times faster and halves the memory traffic, which is the difference between a training run costing one month and three, but the updates the optimizer wants to apply are frequently smaller than the smallest change a half-precision number can represent. This piece is about how that tension is resolved, why the resolution hinges on a detail of how floating-point numbers are laid out, and what changed when the format itself was redesigned for the purpose.\u003c/p\u003e","title":"Mixed Precision — Training in Half the Bits Without Losing the Thread"},{"content":"Almost every language model shares a quiet assumption: to process a single word, it uses all of itself. Every one of its billions of knobs fires for every word, whether that word is a comma or a rare medical term. This feels so natural it barely registers as a choice — but it carries a heavy price. The only way to make such a model know more is to make it bigger, and a bigger model is proportionally slower and costlier, because all of it runs, all the time, for everything. Mixture of Experts — usually shortened to MoE — is the family of models that breaks this bargain. Its idea is to build something vast in what it knows while only switching on a small slice of itself for any given word. This piece is about how that trick works, the surprisingly awkward problems it creates, and the tidy division of labour it does not deliver.\nThe weld between size and cost Start with the problem it solves. An ordinary model is called dense, because all of it is always engaged. In a dense model, capacity and cost are welded together: more knowledge requires more knobs, and more knobs mean more work per word, forever. Picture a factory where making one item means switching on every machine on the floor — including the ones irrelevant to that item. Want more capability? Add machines. But now they all run, for everything, and the electricity bill climbs in lockstep with the capability. There is no way to grow what the factory can do without growing what it costs to run.\nA mixture-of-experts model cuts that weld. Instead of one monolithic machine, it keeps a whole bank of smaller specialists — the experts — and adds a tiny decision-maker — the router — that glances at each incoming word and sends it to just a couple of the most relevant experts. The rest stay switched off for that word. So the model can store an enormous amount of knowledge across all its experts while doing only a small amount of work on any single word. The picture to hold is a large teaching hospital: hundreds of specialist doctors on staff, but each patient is triaged to just the two who fit their case and then goes home. The hospital\u0026rsquo;s total expertise is vast; no one patient consumes it all.\nTwo sizes that get confused The moment a model uses only part of itself, \u0026ldquo;model size\u0026rdquo; splits into two numbers, and keeping them apart is essential. The total parameter count is everything the model stores — every expert, used or not. This is its capacity, the breadth of what it can know; it is the size of the hospital\u0026rsquo;s whole staff. The activated parameter count is the much smaller slice actually put to work on one word — the router plus the two-or-so experts it picked. This is what sets the speed and running cost; it is the handful of doctors a single patient actually sees.\nReal figures make the gap vivid. Mixtral stores roughly 47 billion parameters but activates only about 13 billion per word; DeepSeek\u0026rsquo;s large model stores around 671 billion but works with only about 37 billion at a time. The whole art is buying a large total — more knowledge, better answers — while keeping the activated amount small. A slogan to carry forward: cheap to compute, expensive to store.\nFigure 1. Total parameters against activated parameters.\nA pinch of math: how the router chooses Inside, an expert is nothing exotic — it is just a copy of the ordinary per-word processing stage that already lives in every block, and a mixture-of-experts model simply keeps many such copies, each with its own learned settings. The interesting part is the router, and it is worth a little precision.\nFor a given word, the router takes that word\u0026rsquo;s current representation $x$ and produces a raw score for each of the $N$ experts — one number saying \u0026ldquo;how much does this word want that expert?\u0026rdquo; It does this with a small learned grid $W_g$:\n$$s = W_g \\, x \\qquad \\text{(one score } s_1, \\ldots, s_N \\text{, one per expert)}$$Those scores are turned into proper weights with a softmax , so they are positive and sum to one:\n$$g_i = \\frac{e^{s_i}}{\\sum_j e^{s_j}}$$Now comes the move that saves all the computation: instead of using every expert weighted by $g_i$, the router keeps only the few highest weights — the top-$k$, where $k$ is a small number like one or two — and ignores the rest entirely. Only those chosen experts actually run. Their outputs are then blended back together, each weighted by its gate value:\n$$y = \\sum_{i \\,\\in\\, \\text{top-}k} g_i \\, E_i(x)$$Here $E_i(x)$ is what expert $i$ computes for the word. Everything not in the top-$k$ contributes nothing, because it was never consulted. The triage nurse scores every speciality and picks the best two; the other departments are not consulted. That single \u0026ldquo;keep only the top few\u0026rdquo; step is where a model with a hundred experts still only pays for two.\nFigure 2. Routing one word through a bank of experts.\nThe deep difficulty: learning from a choice Here is the central headache, and it explains most of what makes these models tricky. A model learns by nudging its knobs in whatever direction reduces its mistakes — but the router\u0026rsquo;s action is a choice, and a choice is all-or-nothing. You cannot send a word slightly to expert three. So the smooth learning signal has nothing to grab onto at the moment of selection. The router can be told that the experts it picked worked out and to lean into them, or that they worked out badly and to trust them less, but it never gets a clean signal saying \u0026ldquo;you should have picked a completely different expert\u0026rdquo; — because the expert it did not pick produced nothing to compare against. The router learns to weight its winners; it gets no clean nudge to change its winners.\nThis blind spot breeds a failure called expert collapse. Suppose, by luck at the start, a few experts are slightly better. The router sends them more words; more words means more practice, so they improve further; so the router sends them yet more. Meanwhile the neglected experts get almost no words, never improve, and fall further out of favour. The end state is a handful of experts doing nearly all the work while the rest sit idle — the storage bill for a hundred experts, the capacity of four. In the hospital, every patient goes to the same two doctors and the rest of the staff is paid to be present.\nLoad balancing: gently insisting the work be spread Crucially, the ordinary \u0026ldquo;get the answer right\u0026rdquo; goal does nothing to fix this, because a collapsed model can still give reasonable answers using its few good experts. Balance has to be encouraged separately and deliberately. The fix is to add a second, mild goal alongside correctness: use all the experts roughly evenly. Concretely, we add a small load-balancing penalty to the loss — a term that grows whenever the router is overloading some experts and starving others. If $f_i$ is the fraction of words sent to expert $i$ and $p_i$ is the average gate weight it received, a common form is:\n$$L_{\\text{balance}} = \\lambda \\, N \\sum_{i} f_i \\, p_i$$You do not need to trace the algebra; the intuition is what counts. This term is smallest when the load is spread evenly across all $N$ experts and grows when it piles onto a few, so minimizing it nudges the router to ease off its favourites and give the neglected experts a turn. The dial $\\lambda$ sets how hard this nagging pushes — too gentle and the model collapses anyway, too aggressive and the router is forced to make bad choices just to stay balanced. Tuning this pressure is the single biggest way mixture-of-experts training differs from ordinary training; a dense model has nothing to balance, because it makes no choices.\nFigure 3. Collapse, and the penalty that pushes back.\nTwo practical headaches Real hardware likes fixed, predictable batch sizes, so each expert is given a fixed number of slots per batch — its capacity. But routing is unpredictable, so some experts get swamped and others sit half-empty. If an expert is assigned more words than it has slots, the surplus words are simply dropped — they skip the expert and pass through unchanged, carried along by the rest of the network. A dial sets how much headroom to leave: too little and patients are turned away on busy days, too much and chairs sit empty.\nThe second headache is communication. Because a mixture-of-experts model stores so many experts, they do not fit on one machine and are spread across many. But the words being processed are spread across machines too, and the expert a word needs usually lives on a different machine than the word does. So before the experts run, every machine ships its words out to wherever the right experts are, and ships the results back afterward — every machine talking to every other, twice, per layer. Imagine the hospital split across a dozen buildings, with patients constantly shuttled to the right building and back, until the shuttling, not the doctoring, becomes the slow part. This heavy communication is the defining cost of training these models, and it is why balance matters twice over: a layer finishes only as fast as its busiest expert, so an imbalanced router leaves expensive machines idling at the shuttle stop.\nA reality check, and the bargain that makes it worth it It is tempting to imagine a tidy \u0026ldquo;French expert\u0026rdquo; and a \u0026ldquo;math expert.\u0026rdquo; Reality is less tidy than that. Specialization is real but weak and low-level — experts tend to react to surface patterns like punctuation and word types more than to clean human topics, and responsibility for any subject is usually smeared across several experts. The gains come less from crisp specialists than from sheer extra capacity and the router\u0026rsquo;s freedom to mix a different small combination of experts for every word.\nBut the gains are real, and that is the whole justification for the complexity. At a fixed amount of computing work per word, adding more experts reliably improves the model. In other words, mixture-of-experts converts a storage budget into quality without spending more compute budget. A dense model has one lever — spend more compute, get more of everything, pay for all of it on every word. A mixture-of-experts model has two levers, buying capacity with comparatively cheap storage instead of expensive computation. There is a sweet spot, since the returns shrink and the storage and shuttling are genuine costs, but that two-lever bargain is why the whole enterprise is worthwhile.\nWhat to carry forward A mixture-of-experts model breaks the weld between how much a model knows and how much work it does per word. It replaces the single processing stage with a bank of experts and a tiny router that scores them all, keeps only the top few, and blends their outputs — so total capacity can be enormous while the per-word cost stays small. The price of this cleverness is a cluster of problems rooted in the router\u0026rsquo;s all-or-nothing choices: expert collapse, which forces a deliberate load-balancing penalty; dropped words when experts overflow; and heavy machine-to-machine shuttling. And the defining trade in the end is memory for quality — cheap to compute, expensive to store. What the bargain does not buy is the tidy division of labour the name suggests: specialization is real, weak and low-level, with responsibility for any subject smeared across several experts, so the gain is extra capacity and the router\u0026rsquo;s freedom to mix rather than a bank of specialists.\nThe figures in this article follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/articles/mixture-of-experts/","summary":"\u003cp\u003eAlmost every language model shares a quiet assumption: to process a single word, it uses \u003cem\u003eall\u003c/em\u003e of itself. Every one of its billions of knobs fires for every word, whether that word is a comma or a rare medical term. This feels so natural it barely registers as a choice — but it carries a heavy price. The only way to make such a model know more is to make it bigger, and a bigger model is proportionally slower and costlier, because all of it runs, all the time, for everything. Mixture of Experts — usually shortened to MoE — is the family of models that breaks this bargain. Its idea is to build something vast in what it \u003cem\u003eknows\u003c/em\u003e while only switching on a small slice of itself for any given word. This piece is about how that trick works, the surprisingly awkward problems it creates, and the tidy division of labour it does not deliver.\u003c/p\u003e","title":"Mixture of Experts — A Model That Only Switches On What It Needs"},{"content":"A transformer does not care what its inputs mean. It consumes a sequence of vectors and produces a sequence of vectors, and nothing in attention or the feed-forward layers makes any reference to language. The embedding table is the only component that assumes its inputs are tokens, and it is a lookup rather than a computation. Which means the path to a model that sees is narrower than it looks: if you can turn an image into a sequence of vectors that live in the same space as token embeddings, the rest of the machinery will process it without modification. The interesting question is not whether this works — it plainly does — but what it costs, what gets lost, and why a model that can describe a photograph in detail will confidently misread a clock face. This piece is about the join between a vision system and a language model, and what remains awkward about it.\nImages as sequences The first step is to make an image into something sequence-shaped. The standard answer is to cut it into a grid of square patches — typically fourteen or sixteen pixels on a side — flatten each into a vector, and treat the resulting list as tokens. A 224-pixel image at 14-pixel patches becomes 256 patches, which is a sequence length a transformer handles comfortably.\nThis is tokenization with a different substrate, and it inherits the same character: a fixed, non-learned segmentation that decides what the model can distinguish. It also inherits the same failure mode. A patch grid imposes a resolution floor, and detail finer than a patch is averaged away before the model sees anything. Small text, thin lines, the position of a clock\u0026rsquo;s minute hand relative to its tick marks — these are sub-patch details, and a model that misreads them is not being careless. It received an average.\nThe quadratic cost of attention makes this hard to fix by simply using smaller patches, since halving the patch size quadruples the sequence length. The practical response has been to process a high-resolution image as several crops — a downscaled view of the whole plus a grid of full-resolution tiles — and let the model combine them. This is why the token cost of an image in current systems scales with its dimensions rather than being fixed, and why a dense screenshot can consume several thousand tokens.\nFigure 1. A resolution floor set before the model looks.\nThree ways to join the two halves Given patch vectors, the question is how they meet the language model. Three arrangements have been used, and the field has converged fairly decisively.\nCross-attention. Leave the language model\u0026rsquo;s own sequence alone and add extra attention layers that let text positions attend to image features held separately. The image never enters the text sequence, so it costs no context length, and the language model\u0026rsquo;s original weights can be frozen while only the new layers train. This was the early approach and it works. Its drawback is architectural surgery: the language model must be modified, which means the arrangement does not travel to a new base model without redoing it.\nProjection into the sequence. Run the image through a vision encoder, pass the resulting patch features through a small trained network — often just one or two matrices — that maps them into the language model\u0026rsquo;s embedding space, and insert them into the token sequence as if they were tokens. Nothing about the language model changes. From its point of view it is reading a sequence of vectors, some of which happen to have come from somewhere other than the embedding table.\nResampling. A middle option that uses a small set of learned queries to attend over the patch features and produce a fixed number of output vectors, compressing an arbitrary number of patches into, say, sixty-four. This bounds the context cost regardless of image size, at the price of discarding detail through the bottleneck.\nThe second has won, for a reason worth naming: it requires no modification to the language model at all, so any improvement to the underlying model is inherited immediately, and the connecting network is small enough to train cheaply. The elegance is that the \u0026ldquo;projector\u0026rdquo; is doing something conceptually trivial — a change of coordinates — and the heavy lifting is done by two components that were trained separately for other purposes.\nFigure 2. Three joins, and why the middle one won.\nWhy the vision encoder is usually contrastive The vision half is generally not trained from scratch alongside the language model. It is a pretrained encoder, and the pretraining objective that dominates is contrastive — the same arrangement described under retrieval embeddings , applied across modalities.\nTake a very large collection of images paired with their captions. Encode each image and each caption, and train so that an image is closer to its own caption than to any other caption in the batch. Both encoders learn in tandem, and what results is an image encoder whose outputs are organized by semantic content — because the only thing that predicts a caption is what the image is about.\nThis matters for the join. A vision encoder trained to classify images into a thousand categories learns features useful for those categories. A contrastively trained one has been shaped by natural language descriptions of unrestricted variety, so its representation is already aligned with the kinds of distinction language makes. Bolting it to a language model is a shorter journey. The projector\u0026rsquo;s job is a coordinate change rather than a translation.\nThe connection also runs the other way: diffusion models that generate images from text use the same kind of jointly-trained encoder to condition on a prompt, which is why the same family of models underpins both understanding and generation.\nA pinch of math: the training is the ordinary one The surprising part of the whole arrangement is how little the objective changes.\nWrite the input sequence as a mixture of embedded tokens and projected patches:\n$$h_0 = \\big[\\, e(t_1), \\ldots, e(t_k), \\; P(v_1), \\ldots, P(v_m), \\; e(t_{k+1}), \\ldots \\,\\big]$$where $e$ is the embedding lookup, $v_j$ are patch features from the vision encoder, and $P$ is the projector. The model then runs exactly as before, and the loss is exactly as before — cross-entropy on the next text token:\n$$\\mathcal{L} = -\\sum_{t \\in \\text{text}} \\log p_\\theta(x_t \\mid x_{","permalink":"https://www.fanfani.net/articles/multimodality/","summary":"\u003cp\u003eA transformer does not care what its inputs mean. It consumes a sequence of vectors and produces a sequence of vectors, and nothing in \u003ca href=\"/articles/attention/\"\u003eattention\u003c/a\u003e\n or the \u003ca href=\"/articles/feed-forward-layers/\"\u003efeed-forward layers\u003c/a\u003e\n makes any reference to language. The \u003ca href=\"/articles/embeddings/\"\u003eembedding table\u003c/a\u003e\n is the only component that assumes its inputs are tokens, and it is a lookup rather than a computation. Which means the path to a model that sees is narrower than it looks: if you can turn an image into a sequence of vectors that live in the same space as token embeddings, the rest of the machinery will process it without modification. The interesting question is not whether this works — it plainly does — but what it costs, what gets lost, and why a model that can describe a photograph in detail will confidently misread a clock face. This piece is about the join between a vision system and a language model, and what remains awkward about it.\u003c/p\u003e","title":"Multimodality — Teaching a Language Model to See"},{"content":"Every block in a transformer is preceded by a small operation that rescales its input, and it is the least glamorous thing in the architecture. It computes no features, stores no knowledge, and would be omitted from any diagram drawn on a napkin. It is also the difference between a model that trains and one that does not. The residual stream grows as it rises, because every block adds and nothing subtracts; a block trained to read a signal of one typical size will misbehave when handed one ten times larger; and the gradient inherits every one of those distortions on the way back down. This piece is about the operation that keeps the numbers in range, the variant that discarded half of it and lost nothing, and the placement decision that mattered more than either.\nThe problem with a signal that drifts Consider what a block actually needs from its input. Its weights were set during training under some particular distribution of incoming values, and everything about its behaviour — which activations saturate, how large its outputs are, how sensitive it is to a small change — depends on that distribution holding. If the input arrives at three times the expected magnitude, the block\u0026rsquo;s contribution is roughly three times too large, that contribution is added to the stream, and the next block receives something further out of range still.\nNow stack eighty of these. Small drifts compound multiplicatively, and the top of the tower ends up operating in a regime the bottom never saw. Worse, the drift moves during training: as the early layers\u0026rsquo; weights change, the distribution they emit changes, and every later layer finds that the input it had adapted to has shifted underneath it. Each layer is chasing a target that the layers below keep moving. The condition has a name — internal covariate shift — and while the original diagnosis has been argued over ever since, the practical fact is not in dispute. Unnormalized deep networks are difficult to train, need tiny learning rates, and are exquisitely sensitive to initialization.\nThe gradient suffers symmetrically. A backward pass through a layer scales the gradient by something related to that layer\u0026rsquo;s magnitudes, and eighty such scalings compound into the same vanishing or exploding problem that residual connections were introduced to solve. Residuals fix the path; they do not fix the scale.\nThe fix: standardize before every block The remedy is to insert, before each block, an operation that puts its input into a known state regardless of what arrived. Layer normalization does this by working across the features of a single token: compute the mean and standard deviation of that token\u0026rsquo;s vector, subtract the mean, divide by the standard deviation.\n$$\\hat{x} = \\frac{x - \\mu}{\\sqrt{\\sigma^2 + \\epsilon}}, \\qquad \\mu = \\frac{1}{d}\\sum_i x_i, \\qquad \\sigma^2 = \\frac{1}{d}\\sum_i (x_i - \\mu)^2$$Whatever the block receives now has zero mean and unit variance across its dimensions, always. The drift is removed at the entrance to every block rather than being allowed to accumulate.\nStandardizing everything would be too blunt on its own, because sometimes a layer genuinely wants its output scaled or offset, so the operation ends with a learned rescaling — a gain $\\gamma$ and a bias $\\beta$, one per dimension:\n$$\\text{LayerNorm}(x) = \\gamma \\odot \\hat{x} + \\beta$$The model can therefore undo the normalization if it wants to, but must do so deliberately, one dimension at a time, using parameters that gradient descent controls. The default is standardized; departures are learned.\nIt is worth being clear about what \u0026ldquo;layer\u0026rdquo; norm normalizes over, because the name is unhelpful. It computes its statistics across the features of one token, independently for every token and every sequence. This is what makes it suitable for language, where batches contain sequences of different lengths and where generation happens one token at a time with no batch to speak of. The older alternative, batch normalization, computes statistics across the batch dimension for each feature, which works beautifully for images and is close to unusable here.\nWhat RMSNorm dropped, and why nothing broke Layer normalization does two things: it centres, by subtracting the mean, and it scales, by dividing by the standard deviation. Somebody eventually asked which of the two was load-bearing.\nThe answer, empirically, is the scaling. Drop the centring entirely, drop the learned bias with it, and divide instead by the root-mean-square of the vector:\n$$\\text{RMSNorm}(x) = \\gamma \\odot \\frac{x}{\\sqrt{\\frac{1}{d}\\sum_i x_i^2 + \\epsilon}}$$This is root mean square normalization, and models trained with it match models trained with the full version while spending noticeably less time and memory on the operation — no mean to compute, no subtraction pass, no bias parameters, and a simpler backward pass. Given that normalization runs twice per block for every token in every sequence, the saving is real, on the order of a few percent of total training time.\nWhy the centring turns out to be dispensable is not fully settled, and it is more honest to say so than to invent a reason. The most common suggestion is that re-centring mattered more in the shallower, less carefully initialized networks where layer normalization was first developed, and that in a modern transformer the mean of the stream is close to zero anyway, so subtracting it removes almost nothing. Whatever the explanation, essentially every large model built in recent years uses the simpler version, which makes this one of the cleaner examples of a component being pared back once someone checked which half was doing the work.\nFigure 1. What RMSNorm dropped.\nWhere to put it: the change that mattered most The original transformer placed normalization after each sub-block, applying it to the sum of the input and the block\u0026rsquo;s contribution. The stream would be normalized, passed through attention, added back, and the result normalized again on the way to the next block.\nThis is a worse arrangement than it appears, and the reason is visible in the residual stream picture. Putting a normalization on the residual path means the path is no longer clean: the gradient travelling back down the tower must pass through a normalization at every level, and the express route that made depth trainable is fitted with a gate at every floor. Models built this way needed a long learning-rate warm-up to train at all, were fragile at depth, and could diverge outright if the warm-up was shortened.\nMoving the normalization inside the branch — normalizing the block\u0026rsquo;s input, leaving the residual addition untouched — fixes it. Now the path from the top of the tower to the bottom is pure addition, exactly as intended, and the normalization only ever guards what feeds into a block rather than what flows between them. Written out, the difference is one set of parentheses:\n$$\\text{post-norm:}\\quad x_{\\text{out}} = \\text{Norm}\\big(x + f(x)\\big) \\qquad\\qquad \\text{pre-norm:}\\quad x_{\\text{out}} = x + f\\big(\\text{Norm}(x)\\big)$$Pre-norm models train stably at depths where post-norm models will not train at all, tolerate much shorter warm-ups, and are far less sensitive to the learning rate. Effectively every large model now uses it.\nThere is one honest cost, and it follows from the same property that makes it work. Because nothing on the residual path rescales, the stream\u0026rsquo;s magnitude grows steadily with depth — and since each block reads a normalized copy, a block\u0026rsquo;s relative contribution shrinks as the stream it is being added to gets larger. Very deep pre-norm models can therefore end up with upper layers whose contributions are, proportionally, barely felt. The usual remedy is a single final normalization at the top of the tower before the unembedding, plus initialization schemes that scale each block\u0026rsquo;s output down according to depth so that the accumulated total stays sane. Some recent designs go further and normalize on both sides of each block, recovering a little of the post-norm discipline without putting anything back on the residual path.\nFigure 2. Post-norm gates the residual path; pre-norm does not.\nFigure 3. Stream magnitude grows; each block\u0026rsquo;s share shrinks.\nNormalizing inside attention One further use has become standard and is worth mentioning because it addresses a specific failure rather than a general one.\nThe scores inside attention are dot products of queries and keys, and nothing constrains their magnitude. Over a long training run, some heads develop very large query or key vectors, the scores grow with them, and the softmax saturates — one weight goes to essentially one, the rest to essentially zero, and the gradient through that head vanishes. The head stops learning, and in the worst case the loss spikes and the run is damaged.\nThe fix is to normalize the queries and keys before the dot product, capping how large a score can get regardless of what the projections do. This is QK normalization, it costs almost nothing, and it has become common in large models specifically because it removes one of the more frequent causes of a training run falling over at scale. It is a good illustration of what these operations are for: not improving the model\u0026rsquo;s representations, but keeping the numbers in a range where the rest of the machinery behaves.\nWhat to carry forward Normalization computes nothing and knows nothing; it exists so that everything else can rely on its input being in a predictable range. Without it, magnitudes drift as the residual stream accumulates, every block finds itself reading a signal unlike the one it adapted to, the distribution moves during training as the layers below change, and the gradient inherits all of it. Layer normalization standardizes across the features of a single token — mean zero, unit variance — then applies a learned per-dimension gain and bias so the model can deliberately depart from the standard if it wants to; computing statistics per token rather than per batch is what makes it usable for variable-length sequences and single-token generation. RMSNorm discards the centring and the bias, dividing only by the root mean square, and matches the full version at lower cost, which is why nearly every recent model uses it. The most consequential decision, though, is placement: normalizing the sum after each block puts a gate on the residual path and produces models that need long warm-ups and break at depth, whereas normalizing the input to each block leaves the path pure addition and trains stably far deeper — at the cost of a stream whose magnitude grows with depth, so that later blocks contribute proportionally less, usually managed with a final normalization at the top and depth-aware initialization. And normalizing queries and keys before the attention dot product caps score magnitudes and prevents the softmax saturation that is one of the more common ways a large training run falls over.\nThe figures in this article follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/articles/normalization/","summary":"\u003cp\u003eEvery block in a transformer is preceded by a small operation that rescales its input, and it is the least glamorous thing in the architecture. It computes no features, stores no knowledge, and would be omitted from any diagram drawn on a napkin. It is also the difference between a model that trains and one that does not. The \u003ca href=\"/articles/residual-stream/\"\u003eresidual stream\u003c/a\u003e\n grows as it rises, because every block adds and nothing subtracts; a block trained to read a signal of one typical size will misbehave when handed one ten times larger; and the gradient inherits every one of those distortions on the way back down. This piece is about the operation that keeps the numbers in range, the variant that discarded half of it and lost nothing, and the placement decision that mattered more than either.\u003c/p\u003e","title":"Normalization — Keeping the Signal Steady Enough to Stack"},{"content":"A frontier model does not fit on one machine, and the arithmetic makes the point without ceremony: the article on AdamW \u0026rsquo;s sixteen bytes per parameter puts a four-hundred-billion-parameter model at six and a half terabytes of training state, against perhaps eighty gigabytes on a single accelerator. Something has to be divided. What makes this more than a logistics problem is that there are five fundamentally different things you can divide — the data, the width of each matrix, the depth of the tower, the sequence itself, and, where the model has them, the experts — and each division has a completely different communication bill attached. Getting the combination right is worth a factor of several in training time, and getting it wrong produces a cluster of expensive machines mostly waiting for each other. This piece is about the five axes and how they are stacked.\nThe constraint that governs everything Before the five, the fact that determines all of them. Inside a single machine, accelerators are linked by a dedicated high-speed fabric offering several hundred gigabytes per second and very low latency. Between machines, the link is a network — fast by ordinary standards, and roughly an order of magnitude slower than the fabric inside a box, with far worse latency.\nEvery parallelism strategy is therefore evaluated not by how much communication it requires in the abstract, but by how much it requires per unit of computation, and whether that communication can be confined inside a single machine. A strategy that talks constantly is fine within a box and ruinous across boxes. This one fact explains nearly every layout choice that follows.\nFigure 1. Five axes, five communication bills.\nData parallelism: split the batch The simplest division. Give every device a complete copy of the model and a different slice of the batch. Each computes gradients on its own slice; the gradients are then averaged across all devices, and everyone applies the identical update.\nCommunication happens once per step and involves exchanging one gradient per parameter. That is a large volume but a fixed one — it does not grow with batch size — so it amortizes well when each device is doing substantial work, and it overlaps neatly with computation, since the gradients for the last layers are ready while the earlier layers are still being computed. It is the cheapest strategy per unit of throughput and always the first one reached for.\nIts limitation is that it does nothing about memory. Every device holds the entire model, so it only helps once the model already fits, and for a large model it does not.\nThe refinement that changes this is sharding the training state. Notice that a device does not need the optimizer\u0026rsquo;s running averages for every parameter — it only needs them at the moment it applies the update, and the update for any given parameter could be performed by whichever device happens to own it. So the optimizer state can be split across the data-parallel group, each device owning a slice. Extend the same logic and the gradients can be split too, and then the weights themselves, with each device holding only its shard and gathering the others temporarily when a layer is about to be used. Memory per device falls by roughly the number of devices in the group; the price is more communication, since weights must be gathered before each layer\u0026rsquo;s forward and backward pass rather than held permanently. This progression — state, then gradients, then weights — is the well-known staged scheme, and its final stage is really a hybrid of data and model parallelism wearing data parallelism\u0026rsquo;s interface.\nTensor parallelism: split each matrix Rather than replicating a layer, cut its matrices into pieces and give each device a piece, so that one layer\u0026rsquo;s computation is performed jointly.\nThe way the cuts are chosen is elegant, and worth seeing because it minimizes the talking. Take the feed-forward layer , which expands and then contracts. Split the expanding matrix by columns, so each device produces a distinct slice of the hidden activations — no communication is needed, since each device has everything it requires to compute its slice. Apply the nonlinearity elementwise, which is also local. Now split the contracting matrix by rows, matching the way the hidden state was divided. Each device computes a partial result covering the full output width, and the true output is their sum. One addition across devices, and the layer is done.\nSo a pair of matrices costs exactly one communication rather than two, because the two splits were chosen to compose. Attention is even more natural: distribute whole heads to different devices, since heads are independent by construction, and communicate once when concatenating their outputs.\nThe bill is one all-reduce per sub-block, which is twice per transformer block, on the full activation tensor. That is a great deal of communication, and it sits on the critical path — nothing can proceed until it completes. Tensor parallelism is therefore used within a machine, across the devices sharing the fast internal fabric, and almost never across machines. In practice this caps the group at the number of accelerators in one box, typically eight.\nPipeline parallelism: split the depth Give device one the first ten layers, device two the next ten, and so on. Activations flow forward from one device to the next and gradients flow back.\nCommunication is minimal: only the activations at the boundaries between stages, which is a small tensor compared to anything the other strategies move. This is what makes it the strategy for crossing machine boundaries — it is the only one whose bill is small enough to tolerate a slow link.\nIts problem is idleness. If a batch is fed through as a single unit, device two cannot start until device one has finished, and device one has nothing to do while devices two through eight work their way through. The fraction of time wasted is substantial and grows with the number of stages.\nThe fix is to split the batch into micro-batches and feed them through in sequence, so that once the pipeline is full, every stage is working on a different micro-batch simultaneously. Only the beginning and end of each step have idle stages — the bubble — and its relative size falls as the number of micro-batches rises. With enough micro-batches the waste becomes a few percent. The schedules that interleave forward and backward passes to shrink the bubble further, and to reduce how many activations must be held in flight, are their own small literature.\nFigure 2. Micro-batches fill the pipeline and shrink the bubble.\nSequence and expert parallelism Two more axes, each addressing something the first three do not.\nSequence parallelism splits along the token dimension: each device holds a segment of the sequence. This is the answer to long contexts, where the activations for a single example exceed one device\u0026rsquo;s memory no matter how the model is divided. The difficulty is that attention is precisely the operation that needs every position to see every other, so segments must exchange their keys and values. The standard approach passes them around a ring of devices, so that each segment sees every other\u0026rsquo;s keys in turn while overlapping the transfer with computation. Operations that are per-token — normalization, feed-forward layers — need no communication at all, which makes this cheaper than it first sounds.\nExpert parallelism applies to mixture-of-experts models and places different experts on different devices. Tokens must then travel to wherever their assigned expert lives and their results travel back, which is the all-to-all communication described in that article — twice per sparse layer, with an irregular pattern that depends on the router\u0026rsquo;s decisions and is therefore hard to overlap. It is also the reason load balancing has a throughput consequence rather than merely a quality one: everyone waits for the device holding the most-requested expert.\nA pinch of math: stacking the axes The strategies are not alternatives; they multiply. If $d$, $t$, and $p$ are the data, tensor, and pipeline degrees, the total device count is\n$$N = d \\times t \\times p$$and the global batch is assembled as micro-batch size times gradient accumulation steps times $d$. Choosing $(d, t, p)$ for a given cluster is the practical exercise, and the reasoning follows directly from the communication costs.\nSet $t$ to the number of accelerators inside one machine, and no larger, because tensor parallelism\u0026rsquo;s per-block all-reduce cannot cross a slow link without dominating. Set $p$ to whatever is needed to make the model\u0026rsquo;s parameters fit in $t \\times p$ devices\u0026rsquo; memory, and no larger, because every additional stage enlarges the bubble. Give all remaining devices to $d$, since data parallelism has the best ratio of communication to computation and scales furthest. Then, if activations still do not fit, add sequence parallelism; if the model is sparse, expert parallelism takes the place of some of the tensor degree.\nA worked instance makes it concrete. On a thousand accelerators in machines of eight: $t = 8$ confined within each machine, $p = 16$ spanning sixteen machines with the small boundary transfers crossing the network, leaving $d \\approx 8$ replicas of that whole arrangement, each processing a different slice of the batch. Communication is heaviest exactly where the fabric is fastest, and lightest where it is slowest, which is the entire design goal.\nFigure 3. A thousand accelerators, laid out.\nWhat to carry forward A frontier model\u0026rsquo;s training state is measured in terabytes and a single accelerator holds tens of gigabytes, so the work must be divided — and the choice of division is governed by one fact, that the fabric inside a machine is roughly an order of magnitude faster than the network between machines. Data parallelism replicates the model and splits the batch, communicating one gradient per parameter once per step, which amortizes well and overlaps with computation but does nothing for memory until the training state is sharded across the group in the staged fashion that splits first the optimizer state, then the gradients, then the weights. Tensor parallelism splits the matrices themselves, choosing column-wise and row-wise cuts that compose so a pair of matrices costs a single all-reduce — but that reduce sits on the critical path twice per block, so it is confined within a machine. Pipeline parallelism splits the depth and communicates only boundary activations, which is small enough to cross machines; its cost is the idle bubble, shrunk by dividing the batch into micro-batches that keep every stage busy. Sequence parallelism splits the tokens, which is how long contexts fit, and requires passing keys and values around a ring so that attention can still see everything. Expert parallelism places experts on different devices and pays the all-to-all that makes load balancing a throughput concern. The five multiply rather than compete, and the standard reasoning is to set tensor degree to a machine\u0026rsquo;s device count, pipeline degree to the minimum that makes the model fit, and give everything left to data parallelism — placing the heaviest communication precisely where the interconnect is fastest.\nThe figures in this article follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/articles/parallelism/","summary":"\u003cp\u003eA frontier model does not fit on one machine, and the arithmetic makes the point without ceremony: the article on \u003ca href=\"/articles/adamw/\"\u003eAdamW\u003c/a\u003e\n\u0026rsquo;s sixteen bytes per parameter puts a four-hundred-billion-parameter model at six and a half terabytes of training state, against perhaps eighty gigabytes on a single accelerator. Something has to be divided. What makes this more than a logistics problem is that there are five fundamentally different things you can divide — the data, the width of each matrix, the depth of the tower, the sequence itself, and, where the model has them, the experts — and each division has a completely different communication bill attached. Getting the combination right is worth a factor of several in training time, and getting it wrong produces a cluster of expensive machines mostly waiting for each other. This piece is about the five axes and how they are stacked.\u003c/p\u003e","title":"Parallelism — Splitting a Model Five Different Ways"},{"content":"Position is the detail most accounts of a transformer dispose of in half a sentence, somewhere near the embedding lookup, on the way to the interesting part. That treatment has it backwards. The mechanism at the centre of the architecture is not inattentive to word order but provably indifferent to it, making the dog bit the man and the man bit the dog one object as far as the arithmetic goes, and the fixes for that indifference are design decisions with measurable consequences — in how far a model can read, in how fast it runs, in what it quietly finds hard. A good deal of the difficulty with long inputs is a positional problem in other clothes. This piece is about why order has to be supplied from outside, what each scheme for supplying it has cost, and what none of them delivers.\nAttention treats a sentence as a set Attention works entirely by comparison. Every position emits a query and a key, each query is dotted against every key, and those scores become the weights deciding whose value gets pulled in. Read that description again and look for the part that mentions where a token sits. There is nothing. The score between two positions is a function of their two vectors and of nothing else, so shuffling the tokens of a sentence leaves the set of comparisons unchanged — the same pairs, the same scores, dealt into different slots. The property has a name. The mechanism is permutation-equivariant: permute the input and the output is permuted identically, with no other difference.\nFor a device meant to process unordered collections this is exactly right. For a model of language, where the difference between two orderings of five words is frequently the whole of the meaning, it is a startling place to begin.\nThe indifference was not an oversight, which is why it cannot simply be repaired. Permutation equivariance is the same property as parallelism: because no position\u0026rsquo;s computation depends on the order positions arrive in, all of them can be computed at once, which is what let the architecture be trained on hardware built for doing many identical things simultaneously, and what the recurrent designs it replaced could not do. Order was not left out of attention by accident. It was removed, deliberately and profitably, and then put back by other means. Every scheme below is a patch, and the patch is what parallelism costs.\nThe first schemes stamped position onto the input The original transformer, from 2017, took the most direct route: if the mechanism cannot see position, write position into what it does see. Before the first layer, each token\u0026rsquo;s embedding has a position-dependent vector added to it, built from sines and cosines at geometrically spaced frequencies — some dimensions cycling every few positions, others taking thousands to come around once.\nThe appeal was considerable and largely theoretical. The encoding is a fixed function, so it adds no parameters and needs no training. It is defined for every integer, so nothing runs out. And it carries an elegant algebraic property: shifting position by a fixed amount is a fixed linear transformation of the encoding, so relative offsets are in principle recoverable.\nThe costs took longer to surface, and there are three. The signal is added to the vector that carries meaning, so position and content compete for the same dimensions of the residual stream . It is injected once, at the bottom, and must survive every layer intact enough to be usable at layer sixty. And the extrapolation the fixed functional form seemed to promise never arrived, because a well-defined encoding for an unseen position is not the same thing as a learned response to one.\nLearned absolute encodings traded the last of these away entirely. Rather than compute the vector, store one trainable vector per position and let the optimizer decide what each seat means. In-distribution this works slightly better, since nothing forces position twelve to relate to position thirteen in any particular way. Outside the training length it fails absolutely rather than gradually: position four thousand and one has no row, and there is nothing sensible to put there. Not a degradation. A wall.\nA pinch of math: what it means to depend on the gap Write the attention score between a query at position $m$ and a key at position $n$ as $s(m,n)$. What every scheme here is reaching for is that the score should depend on the two positions only through their difference:\n$$s(m, n) = h(x_m,\\, x_n,\\, m - n)$$The reason is linguistic rather than mathematical. The relations a model needs — the adjective before this noun, the subject at the head of this clause, the bracket that opened this one — are offsets, and they mean the same thing on page one as on page ninety. A scheme parameterized by absolute index encodes what the model does not want and relies on it to reconstruct what it does.\nThe sinusoidal scheme does not deliver that form, but it comes close enough to be instructive. Take one pair of dimensions driven at frequency $\\omega_i$, so the encoding at position $m$ restricted to that pair is\n$$p_m^{(i)} = \\begin{pmatrix} \\sin(m\\omega_i) \\\\ \\cos(m\\omega_i) \\end{pmatrix}$$Move forward $k$ positions and the angle-addition identities give\n$$p_{m+k}^{(i)} = \\begin{pmatrix} \\cos(k\\omega_i) \u0026 \\sin(k\\omega_i) \\\\ -\\sin(k\\omega_i) \u0026 \\cos(k\\omega_i) \\end{pmatrix} p_m^{(i)}$$which is a rotation through $k\\omega_i$ — a matrix depending on the offset $k$ and not at all on where we started. A shift in position is a fixed linear map on the encoding, the same map everywhere. The bare dot product of two encodings is cleaner still, collapsing to a sum of $\\cos((m-n)\\omega_i)$ across frequencies, a function of the gap alone.\nHere is where property and practice part company. That dot product is not the attention score. The encoding was added to the content, so the vector entering the query projection is $x_m + p_m$, and the score expands into four terms: content against content, content against position, position against content, position against position. Only the last is a clean function of the gap, and even it arrives filtered through two learned matrices with no obligation to preserve the rotation. So the relative offset is linearly recoverable, and that is all. A map exists that would extract it; nothing compels the weights to become that map. The model must learn to use the offset from the lengths it happened to see, and the distance between an algebraic property and an exploited one is where the extrapolation went.\nFigure 1. A shift in position is one rotation.\nMoving the position inside the score If the score is what should depend on the gap, the direct approach is to intervene on the score. Relative schemes do that: compute the content term as usual, then add a learned bias determined by the offset between the two positions, with distant offsets sorted into shared buckets so the count stays finite. This is the right idea, and was recognized as such early. It is also expensive in an awkward way.\nThe expense is structural rather than arithmetic. A relative bias enters inside the score computation — an offset-dependent term per pair of positions, per head, per layer, growing with the square of the length in exactly the place where nothing should. Worse, it requires indexing into the score matrix, the object the fused implementations under FlashAttention exist to avoid writing down. A positional scheme that obstructs the fastest available kernel is not merely slower. It is usually not adopted.\nThe simplest form survived on those grounds. Drop the learning and subtract a penalty proportional to the distance between the two positions, with a fixed slope per head, so some heads look narrowly and others far. It costs almost nothing, it is defined at any length, and models using it degrade gently past their training window. The catch is written into the mechanism: a monotone penalty on distance makes far-away tokens systematically harder to see, whatever they contain. As a prior over language that is broadly correct, since most dependencies are local. As a policy for finding one sentence buried in a hundred pages it is wrong by construction.\nRotary embeddings are the arrangement that won, and they win by turning the query and key vectors themselves, each by an angle proportional to its position, so that when the two are compared only the difference of the angles survives. The treatment is in the article on rotary embeddings . Why it displaced the others is the point here — relative behaviour from absolute bookkeeping, no parameters to store or exhaust, and an operation applied to the two vectors before any score exists, which leaves the fused kernels undisturbed.\nFigure 2. Where each scheme injects position.\nFigure 3. A monotone penalty is the wrong prior for one buried fact.\nNone of them extrapolates The shared failure deserves bluntness, because the schemes are often ranked as though one had solved it. A model trained to four thousand positions does not work at forty thousand under any encoding. What it produces instead is attention that flattens or collapses, and text degrading in a way resembling confusion rather than arithmetic failure. The interventions that genuinely extend a window — rescaling the frequencies so a longer range maps onto angles the model has seen, or adjusting the base rate — are corrections applied with a little further training, as set out under long context . They are repairs, not properties the encoding possessed. The functional form guarantees a defined value at position forty thousand and one. It guarantees nothing about the response.\nTwo further findings sit underneath this, fixed by none of it. Models attend unevenly across position regardless of scheme, favouring the beginning and the end of an input over the middle, an asymmetry owing as much to the softmax normalizing over an ever-larger candidate set as to any encoding. And measured attention decays with distance faster than the encoding alone accounts for, which suggests the locality is partly learned from data in which the useful thing is usually nearby. The encoding supplies a coordinate system. What a model does with the far end of it is a separate question, largely empirical.\nWhat to carry forward Attention compares every query with every key on identical terms, nothing in that comparison refers to where a token sits, and shuffling a sentence leaves the set of comparisons untouched — the mechanism is permutation-equivariant, which for a model of language is a remarkable starting point and is also what makes every position computable at once. Order was removed to buy parallelism, then reinstated by other means, and each means charges for it. Fixed sinusoidal encodings cost no parameters but share the residual stream with meaning, must survive every layer after one injection at the bottom, and do not extrapolate despite an algebra suggesting they should. Learned per-position vectors do slightly better inside the training length and hit a hard wall beyond it, an unseen position having no entry and no substitute. Relative schemes encode the right quantity — the offset rather than the index — and pay for it inside the score computation, where cost grows with the square of the length and the fused kernels are obstructed; the cheap surviving variant, a fixed penalty on distance per head, buys graceful behaviour at new lengths by making distant tokens systematically harder to see, a sound prior for language and the wrong one for finding a fact in a long document. Rotary embeddings prevailed by delivering relative behaviour through absolute bookkeeping, adding no parameters, and acting on the query and key rather than the score, leaving the fast implementations available. The ranking is about cost and compatibility rather than capability. None of these schemes lets a model work at a length it never trained on; the extensions that appear to are rescalings applied with further training, and the uneven, distance-decaying way models attend across a long input is not something a coordinate system can fix.\nThe figures in this article follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/articles/positional-encoding/","summary":"\u003cp\u003ePosition is the detail most accounts of a transformer dispose of in half a sentence, somewhere near the embedding lookup, on the way to the interesting part. That treatment has it backwards. The mechanism at the centre of the architecture is not inattentive to word order but provably indifferent to it, making \u003cem\u003ethe dog bit the man\u003c/em\u003e and \u003cem\u003ethe man bit the dog\u003c/em\u003e one object as far as the arithmetic goes, and the fixes for that indifference are design decisions with measurable consequences — in how far a model can read, in how fast it runs, in what it quietly finds hard. A good deal of the difficulty with long inputs is a positional problem in other clothes. This piece is about why order has to be supplied from outside, what each scheme for supplying it has cost, and what none of them delivers.\u003c/p\u003e","title":"Positional Encoding — Putting Order Back Into a Mechanism That Ignores It"},{"content":"Two of the better-known methods in post-training define themselves against this one: DPO exists to avoid it, and RLVR is it with the reward model replaced by a checker. The thing itself is worth describing directly. Proximal policy optimization arrived from robotics, where the problem was teaching a simulated body to walk without the training process destroying itself, and it turned out to transfer to language with remarkably little modification. The core difficulty it solves is the same in both settings and is worth stating before any of the machinery: in reinforcement learning, unlike in supervised learning, the model generates its own training data, so a bad update does not merely produce one bad step — it changes what the model will see next, and a policy that has degraded will collect worse experience, from which it will degrade further. Supervised learning is forgiving because the dataset does not move. This piece is about how to take the largest step you safely can when it does.\nWhy the setup is harder than it looks Line up what makes this different from ordinary training.\nThere is no correct answer to imitate. Nobody wrote down the response the model should have produced; there is only a number saying how good the one it produced was. The gradient therefore cannot say \u0026ldquo;you should have said this\u0026rdquo; — only \u0026ldquo;what you said was worth 0.7, and here is how to make things like it more or less likely.\u0026rdquo;\nThe reward arrives at the end. A response of five hundred tokens receives one number, and every one of those tokens has to be assigned some share of the credit. This is the credit assignment problem, and it is the reason a naive approach barely works.\nThe data is generated by the policy being trained. Every update changes the distribution of the next batch. This is what makes the process capable of collapsing, and the collapse can be sudden.\nAnd the reward is a proxy — a learned model of human preference, which as discussed under DPO can be satisfied without the underlying preference being satisfied. Optimizing it hard is not desirable even when it is possible.\nThe building blocks The classical answer assembles four pieces, and the ordering below is roughly the order in which each becomes necessary.\nPolicy gradient. The foundational idea is that you can improve an expected reward by increasing the log-probability of actions in proportion to how much reward followed them. Take samples, weight each token\u0026rsquo;s log-probability by the reward the sequence earned, ascend. This is correct in expectation and hopeless in practice, because the variance is enormous: two nearly identical responses can receive rewards of 0.9 and 0.2, and the update is dominated by noise.\nA baseline. Most of that variance is not about the action taken but about the situation. Some prompts are easy and everything scores well; some are hard and nothing does. Subtracting a prediction of what this situation was expected to yield removes that component without changing the average update. What remains is the advantage: how much better this response was than expected, rather than how good it was.\nThat prediction has to come from somewhere, and in classical PPO it comes from a critic — a second network, typically initialized from the same model and comparable in size, trained alongside the policy to predict expected return from any partial sequence. This is the piece that doubles the memory of training, the piece DPO removes by deriving its objective differently, and the piece GRPO removes by using the mean reward of a group of samples to the same prompt as the baseline instead.\nAdvantage estimation over a trajectory. With a critic in hand, the advantage at each token can be estimated in ways that trade bias against variance — from a purely local estimate, which is low-variance but relies entirely on the critic being accurate, to using the full observed return, which is unbiased but noisy. Generalized advantage estimation interpolates between them with a single parameter, and in language-model settings it is usually set near the low-bias end, since the reward genuinely arrives only at the end and there is little intermediate signal for the critic to be right about.\nFigure 1. Four models in memory, and who removes which.\nFigure 2. From reward to advantage to a bounded step.\nA pinch of math: the clipped objective The remaining problem is the step size, and this is where PPO earns its name.\nHaving collected a batch of responses, you would like to extract as much learning from them as possible — several passes of gradient updates rather than one, since generating them was expensive. But those responses were sampled from the policy as it was before any updates. After a few passes the policy has moved, and the samples no longer represent what it would do. Continuing to update on them is drawing conclusions from stale evidence, and it is the classic way for a policy-gradient method to destroy itself.\nThe correction is importance weighting: scale each token\u0026rsquo;s contribution by how much more likely the current policy makes it than the sampling policy did.\n$$\\rho_t = \\frac{\\pi_\\theta(a_t \\mid s_t)}{\\pi_{\\theta_{\\text{old}}}(a_t \\mid s_t)}$$This is unbiased and it is also the source of the instability, because when $\\rho$ becomes large the update becomes enormous — precisely for the tokens the policy has already moved furthest on, which are the ones the evidence supports least.\nPPO\u0026rsquo;s contribution is a blunt and effective fix. Clip the ratio to a narrow band around one, and take the more pessimistic of the clipped and unclipped objectives:\n$$\\mathcal{L} = \\mathbb{E}\\Big[\\, \\min\\big(\\rho_t \\hat{A}_t,\\ \\ \\text{clip}(\\rho_t, 1-\\epsilon, 1+\\epsilon)\\, \\hat{A}_t \\big) \\Big]$$with $\\epsilon$ usually around $0.2$. Follow it through in both cases and the behaviour is exactly what you want. Where the advantage is positive and the policy has already raised that token\u0026rsquo;s probability past $1+\\epsilon$, the clipped term is smaller, the minimum selects it, and the gradient is zero — no further reward for pushing something you have already pushed enough. Where the advantage is negative and the probability has already fallen below $1-\\epsilon$, the same thing happens in reverse. But if the ratio moves the wrong way — the policy made a good token less likely — the unclipped term is selected and the full correction applies, because there is no reason to limit a step that is fixing a mistake.\nThe result is a method that permits multiple passes over each expensive batch of samples while structurally refusing to move too far on any one of them. No trust region has to be computed, no constrained optimization solved. It is one clip and one minimum, and its ratio of practical value to conceptual complexity is about as high as anything in the field.\nFigure 3. The clipped objective, positive and negative advantage.\nThe language-model specifics Applied to text, several additions become standard.\nThe KL penalty to the reference model is the leash, and as the article on KL divergence explains it is used in the mode-seeking direction deliberately: the policy is forbidden from putting mass where the starting model saw none, while being free to narrow. It is applied per token as a subtraction from the reward rather than as a term in the loss, which lets the advantage machinery handle it. Its coefficient is the single most consequential dial in the setup, and some implementations adapt it during training to hold the observed divergence near a target.\nWhitening the advantages within a batch — subtracting the mean and dividing by the standard deviation — is close to universal and matters more than its triviality suggests, since reward models produce scores on an arbitrary scale that drifts as training proceeds.\nValue function initialization is a real difficulty. The critic starts knowing nothing, so its early predictions are poor, so early advantages are mostly noise, at exactly the moment the policy is most impressionable. The usual mitigations are to warm up the critic before updating the policy, or to initialize it from the reward model, which at least has a notion of what good looks like.\nAnd reward hacking remains the standing failure mode, arriving in the specific forms that article described: length inflation, formulaic structure, sycophancy. The KL penalty limits it; it does not prevent it.\nWhere it stands now PPO is more expensive and more finicky than the alternatives — four models in memory, a sampling loop inside the training loop, and a dozen interacting dials — and for a period it looked as though DPO had made it obsolete. That is not how it turned out.\nTwo things kept it in use. It is genuinely on-policy, learning from what the current model actually produces rather than from a fixed dataset, which is the property DPO gives up and the one that matters most as training proceeds and the policy drifts from the data. And when the reward is a program rather than a learned model, the objection that motivated much of the retreat from RL simply evaporates — a checker cannot be hacked in the way a reward model can, so the leash can be loosened and the optimization pushed hard. That is exactly the situation of verifiable rewards , and it is why reinforcement learning came back into the centre of post-training just as it seemed to be leaving. What changed is mostly the critic: GRPO\u0026rsquo;s group-relative baseline removed the most expensive component while keeping the clipped objective intact, so what runs today at scale is recognizably PPO with one piece taken out.\nWhat to carry forward Reinforcement learning on a language model is harder than supervised training because there is no correct answer to imitate, the reward arrives only at the end of a long response, and the model generates its own training data — so a bad update degrades the experience collected next, and the failure can compound. The machinery builds up in stages: policy gradient raises the log-probability of tokens in proportion to the reward that followed, which is correct and far too noisy; a baseline subtracts what the situation was expected to yield, converting reward into advantage and removing most of the variance; and in classical PPO that baseline comes from a critic network the size of the model, which is the component both DPO and GRPO were built to eliminate. The method\u0026rsquo;s own contribution is the clipped objective: importance-weight each token by how much more likely the current policy makes it than the policy that sampled it, then clip that ratio to a narrow band and take the pessimistic branch, so that a token already pushed far enough yields no further gradient while a correction in the wrong direction is still applied in full. That single device allows several update passes over each expensive batch without any one of them moving too far. In language settings it is joined by a per-token KL penalty to the reference model — the leash, in the mode-seeking direction — advantage whitening, and careful critic initialization. It looked superseded by DPO\u0026rsquo;s simplicity, and it was not: it is genuinely on-policy, and when the reward is a verifier rather than a learned proxy the case against hard optimization disappears, which is why what runs at scale today is this method with the critic removed.\nThe figures in this article follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/articles/ppo/","summary":"\u003cp\u003eTwo of the better-known methods in post-training define themselves against this one: \u003ca href=\"/articles/dpo/\"\u003eDPO\u003c/a\u003e\n exists to avoid it, and \u003ca href=\"/articles/rlvr/\"\u003eRLVR\u003c/a\u003e\n is it with the reward model replaced by a checker. The thing itself is worth describing directly. Proximal policy optimization arrived from robotics, where the problem was teaching a simulated body to walk without the training process destroying itself, and it turned out to transfer to language with remarkably little modification. The core difficulty it solves is the same in both settings and is worth stating before any of the machinery: in reinforcement learning, unlike in supervised learning, the model generates its own training data, so a bad update does not merely produce one bad step — it changes what the model will see next, and a policy that has degraded will collect worse experience, from which it will degrade further. Supervised learning is forgiving because the dataset does not move. This piece is about how to take the largest step you safely can when it does.\u003c/p\u003e","title":"PPO — Learning From a Judge Without Believing It Too Much"},{"content":"Ask why language models are trained to predict the next word and the answers tend toward the grand: that language is produced sequentially, that prediction is compression and compression is understanding, that anticipating what comes next is what a mind does. None of that is exactly wrong, and none of it is the reason. The objective won a competition against several serious rivals, on grounds that were largely accounting — how much learning can be squeezed out of each token of text, and out of each unit of compute. For a couple of years around 2018 the favourite was something else entirely. Why the sums came out as they did explains a good deal that otherwise looks arbitrary: why encoder models still beat decoders at certain jobs, why a model is never trained in the situation it is deployed in, and why a handful of losing objectives survive in particular corners. This piece is about the constraint that forced the choice, the arithmetic that settled it, and what the field gave up in exchange.\nThere are no labels, and nobody is going to annotate a trillion tokens Every supervised method needs a target to compare its output against, since gradient descent only reduces a discrepancy between prediction and answer. The text required to train a competent language model runs to trillions of tokens. Those two facts do not sit together. No conceivable annotation effort produces labels at that scale, and if one did, the labels would encode whatever narrow task the annotators were briefed on rather than the general competence wanted.\nSo the objective has to manufacture its own supervision out of the raw material. This is self-supervision, and the trick in every version is the same: hide part of the input, and require the model to reconstruct it from what remains. The text becomes both question and answer. No human is involved, the signal is limited only by the supply of documents, and assembling a corpus becomes a matter of filtering rather than labelling.\nThat leaves a design space with two axes, which is where the real argument lives: what to hide, and how much. Every answer defines a different task, and every task is a proxy — a word doing serious work in that sentence. You are not training the thing you want. You are training on a task whose solution you hope requires it, and that gap is never closed by argument, only by results.\nDensity is what settled the argument The rival formulation, for a while the more respectable one, was masked language modelling: replace roughly fifteen percent of the tokens with a placeholder, let the model read the mutilated sequence in both directions, and ask it to recover what is missing. It produced the encoder models that dominated benchmark tables for several years. It lost anyway, for two reasons worth separating.\nEvery position is a training example. A causal model reads a sequence of a thousand tokens and is scored at all thousand positions — the second token given the first, the third given the first two, and so on to the end. Each is a real prediction with a real target, each contributes a term to the loss, and all come out of one forward pass, because the causal mask stops any position seeing its own successor while leaving it free to see everything before. Nothing is wasted. The masked objective takes a gradient only at the corrupted positions, fifteen tokens in every hundred; the other eighty-five are read, processed at full cost, and scored not at all. Six or seven times less signal for the same compute.\nThe obvious repair does not work, which is what makes this decisive rather than incidental. Masking more would raise the density, but the masked positions are predicted from the unmasked ones, and past some fraction there is not enough context left. The task destroys its own premises. Fifteen percent is roughly where the trade bottoms out. Only by sweeping the corruption rate across the whole range, up to and including everything, does the objective turn into something else — the manoeuvre diffusion language models rest on, and it took years to make it pay.\nGeneration comes free. A model trained to predict the next token is already a generator: sample from its output distribution, append the result, run it again. There is nothing to add. A bidirectional model has no such property, never having been asked to produce a token at a position it could not see, so generating from it needs a procedure bolted on afterwards, and every such procedure is awkward. The decoding machinery everything now depends on exists because the objective handed it over at no cost. That was not the argument in favour at the time. It became the argument that mattered.\nFigure 1. Signal per forward pass: 1,000 targets against 150.\nA pinch of math: the chain rule, and one term per position The reason the causal objective is so tidy is that it is not an approximation of anything. Take a sequence $x_1 \\ldots x_n$ and write its joint probability by repeatedly conditioning:\n$$p(x) = \\prod_{t=1}^{n} p(x_t \\mid x_{","permalink":"https://www.fanfani.net/articles/pretraining-objectives/","summary":"\u003cp\u003eAsk why language models are trained to predict the next word and the answers tend toward the grand: that language is produced sequentially, that prediction is compression and compression is understanding, that anticipating what comes next is what a mind does. None of that is exactly wrong, and none of it is the reason. The objective won a competition against several serious rivals, on grounds that were largely accounting — how much learning can be squeezed out of each token of text, and out of each unit of compute. For a couple of years around 2018 the favourite was something else entirely. Why the sums came out as they did explains a good deal that otherwise looks arbitrary: why encoder models still beat decoders at certain jobs, why a model is never trained in the situation it is deployed in, and why a handful of losing objectives survive in particular corners. This piece is about the constraint that forced the choice, the arithmetic that settled it, and what the field gave up in exchange.\u003c/p\u003e","title":"Pretraining Objectives — Why Predicting the Next Word Was Enough"},{"content":"LoRA removed most of the price of adapting a giant model: only a small pair of narrow grids trains, and the original stays frozen. That collapses the training bookkeeping to almost nothing. One obstacle survives it. Training a small set of new knobs still requires holding the entire frozen model in memory throughout — all those billions of original numbers have to sit there so the calculations can flow through them. For the largest models that alone can demand hundreds of gigabytes, far beyond a single ordinary graphics card. QLoRA — Quantized Low-Rank Adaptation — is the trick that knocks down this last wall. The idea is direct: if the frozen model is what consumes the memory, store the frozen model in a more compact form. This piece is about how the compression works, why the rounding survives it, and what the saving does not reach.\nWhere the memory actually goes To see the fix, first see the problem clearly. Every knob in a model is a number, and a number has to be stored in some number of bits. By default, models are trained using numbers that each occupy sixteen bits — a good balance of range and precision. Multiply sixteen bits by tens of billions of knobs and you arrive at those hundred-gigabyte memory bills. LoRA does nothing to change this, because the frozen base still has to be present in full during the forward pass. So the question becomes: do we truly need sixteen bits per knob just to read the model, or could we get away with fewer?\nFigure 1. Sixteen bits to four, and where the rest goes.\nQuantization: describing each number in fewer bits Storing a number in fewer bits is called quantization, and the intuition is everyday. Think of describing people\u0026rsquo;s heights. You could record each one to the millimetre — precise but bulky. Or you could round everyone to the nearest of a small set of buckets — \u0026ldquo;short, medium, tall, very tall\u0026rdquo; — which loses fine detail but takes far less room. Quantization does exactly this to a model\u0026rsquo;s knobs: instead of sixteen bits of precision per number, it snaps each one to the nearest value in a small, agreed-upon set, and stores only which bucket it landed in. QLoRA pushes this down to just four bits per knob — sixteen possible buckets — cutting the memory for the frozen model to roughly a quarter of its former size. A model that needed a small cluster suddenly fits on a single card.\nThe obvious worry is that rounding every knob so aggressively would wreck the model\u0026rsquo;s carefully learned intelligence. QLoRA\u0026rsquo;s craft is in rounding cleverly enough that it barely does. Three ideas do the heavy lifting.\nThree refinements that keep the intelligence intact The first is choosing the buckets to match how the numbers are actually distributed. A model\u0026rsquo;s knobs are not spread evenly; they cluster tightly around zero, with only a few large values out at the edges. Spacing the buckets evenly would waste most of them on empty territory and leave the crowded middle poorly resolved. QLoRA instead uses buckets spaced so that each one holds roughly the same share of the knobs — fine-grained where the values are dense, coarse where they are sparse. This scheme, built to suit the bell-shaped spread that weights naturally follow, is what lets four bits capture so much more fidelity than four bits normally would.\nThe second refinement recognizes that quantization is done in small blocks, and each block needs a scaling number recorded so its buckets can be sized correctly. With many small blocks, those scaling numbers start to add up to real memory of their own. QLoRA\u0026rsquo;s answer is to apply the method to itself: it quantizes the scaling numbers too, compressing the compression bookkeeping. Small on its own, this \u0026ldquo;double\u0026rdquo; step trims meaningful overhead across a whole model.\nThe third refinement is a memory-management trick for the training run itself. During fine-tuning, occasional spikes in memory demand can overflow a card and crash everything. QLoRA lets these overflows spill into ordinary system memory and flow back when the pressure passes — a pressure-release valve that keeps a run from dying at its most demanding moments, at the cost of a little speed.\nFigure 2. Buckets placed where the weights actually are.\nThe crucial split: store in four bits, compute in sixteen Now the central subtlety, and the piece that makes the whole scheme honest. The four-bit form is used only for storage. The moment the model actually needs to compute with a chunk of its knobs, QLoRA temporarily expands that chunk back up to sixteen-bit numbers, does the math at full precision, and then discards the expanded copy. Storage is cheap and permanent; the precise form is expensive but fleeting, conjured only for the instant it is needed and never held all at once.\n$$W_{\\text{4-bit}} \\;\\xrightarrow{\\text{expand on demand}}\\; W_{\\text{16-bit}} \\;\\xrightarrow{\\text{used in}}\\; W_0 + BA$$Meanwhile the LoRA adapters — the little skinny grids $A$ and $B$ described above — are kept in full sixteen-bit precision the whole time, because they are the part that is actually learning and they are tiny enough to afford it. So the model\u0026rsquo;s vast frozen knowledge sits compressed in four bits, decompressed in small pieces only as calculations sweep through it, while the small, precious, trainable part stays sharp. The frozen giant is stored cheaply; the learning happens in full fidelity on the side.\nFigure 3. Stored in four bits, computed in sixteen.\nWhy the rounding does not ruin the result Rounding a model this hard and still fine-tuning it to near-full quality looks implausible. Two things carry it. First, the refinements above mean the four-bit copy is a genuinely faithful likeness of the original — close enough that computations flowing through it come out nearly unchanged. Second, and more subtly, the errors introduced by rounding the frozen base are exactly the kind of thing the trainable adapters can absorb. Because $A$ and $B$ are learning in full precision against the real task, they naturally adjust to compensate for the small distortions of the quantized base, steering the final result back on course. The precise, learning part absorbs the imprecision of the frozen one. The upshot, demonstrated repeatedly in practice, is that QLoRA fine-tuning reaches quality indistinguishable from full sixteen-bit LoRA — while fitting a model many times larger onto the same hardware.\nWhat it changed The practical consequence is hard to overstate. Fine-tuning the very largest open models had been the preserve of organizations with serious clusters. QLoRA brought it down to a single high-end graphics card of the sort a hobbyist or small lab might own. It democratized adaptation: the combination of a frozen, four-bit base and a handful of full-precision LoRA adapters means almost anyone can take a state-of-the-art model and teach it their own speciality over a weekend. It is LoRA\u0026rsquo;s insight — freeze the giant, train a sliver — carried to its logical end by refusing to pay full price for storing the giant at all.\nWhat to carry forward QLoRA removes the last barrier that plain LoRA left standing: the need to keep the whole frozen model in memory at full precision. It quantizes that frozen base down to four bits per knob — storing each number as the nearest of sixteen buckets, chosen to match how weights are really distributed — which shrinks the model to roughly a quarter of its size. The four-bit form is used only for storage; each chunk is briefly expanded back to full precision exactly when it is needed for computation, then thrown away. The small LoRA adapters stay in full precision and do all the learning, and because they train against the real task they quietly absorb the rounding errors of the compressed base. The result is fine-tuning of the largest models on a single graphics card, at quality that rivals the far more expensive full-precision approach. What it buys is access to the training stage and nothing beyond it: the served model still has to be quantized on its own terms, as the article on inference quantization sets out, and the frozen base is as frozen as it was under plain LoRA, so the same constraint on what an adapter can move applies here too, on cheaper hardware.\nThe figures in this article follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/articles/qlora/","summary":"\u003cp\u003e\u003ca href=\"/articles/lora/\"\u003eLoRA\u003c/a\u003e\n removed most of the price of adapting a giant model: only a small pair of narrow grids trains, and the original stays frozen. That collapses the \u003cem\u003etraining\u003c/em\u003e bookkeeping to almost nothing. One obstacle survives it. Training a small set of new knobs still requires \u003cem\u003eholding the entire frozen model in memory\u003c/em\u003e throughout — all those billions of original numbers have to sit there so the calculations can flow through them. For the largest models that alone can demand hundreds of gigabytes, far beyond a single ordinary graphics card. QLoRA — \u003cem\u003eQuantized\u003c/em\u003e Low-Rank Adaptation — is the trick that knocks down this last wall. The idea is direct: if the frozen model is what consumes the memory, store the frozen model in a more compact form. This piece is about how the compression works, why the rounding survives it, and what the saving does not reach.\u003c/p\u003e","title":"QLoRA — Fine-Tuning a Giant on a Single Graphics Card"},{"content":"A model\u0026rsquo;s knowledge is fixed at the end of training, distributed across billions of weights in a form nobody can inspect, edit, or attribute. It cannot tell you where it learned something. It cannot be told that a fact has changed without retraining. And when it does not know an answer it will frequently produce one anyway, because nothing in its training taught it to distinguish confident recall from confident invention. Retrieval-augmented generation addresses all of this by an intervention that is almost embarrassingly direct: before asking the model to answer, go and find the relevant documents, and put them in the prompt. Nearly all the difficulty, however, lives in the retrieval step and in the seam between the two systems, and the naive version fails in ways that are entirely predictable once that seam is understood. This piece is about where the pipeline actually breaks, what the seam demands of both halves, and when retrieval is the wrong instrument to reach for.\nParametric and non-parametric knowledge The framing worth adopting is that a system of this kind has two kinds of memory.\nParametric knowledge is what the weights hold. It is fast, requires no lookup, generalizes across phrasings, and supports inference — the model can combine things it knows in ways nobody wrote down. It is also unverifiable, frozen at training time, impossible to edit reliably, and blends smoothly into fabrication at its edges.\nNon-parametric knowledge is what sits in the context window. It is current as of retrieval, citable, editable by editing a document, and confined to whatever was actually fetched. It costs context, it costs the cache , and it is only as good as the search that produced it.\nThe two are complements. The model supplies language, reasoning, and the ability to synthesize; the retrieved documents supply the facts. The most common failure of intuition about these systems is to expect the retrieved documents to fix a reasoning problem, or to expect the model to fix a retrieval problem. Neither happens.\nFigure 1. Parametric and non-parametric knowledge.\nThe pipeline, and where it actually breaks The standard arrangement has four stages, and it is worth walking them in order because the failure modes distribute very unevenly.\nChunking. Documents are split into passages, because retrieving a whole book is useless and retrieving a sentence loses its context. This unglamorous step causes a startling share of real failures. Split too small and a passage lacks the context to be interpretable — a paragraph beginning \u0026ldquo;This does not apply to contractors\u0026rdquo; is worse than useless without knowing what this is. Split too large and a passage contains one relevant sentence surrounded by noise that dilutes both its retrieval score and the model\u0026rsquo;s attention. Splitting on structure rather than on a fixed token count helps, as does prepending each chunk with its document title and section heading so it can stand alone.\nIndexing and retrieval. Each chunk is turned into a vector by an embedding model and stored; a query is embedded the same way and the nearest chunks are returned. This is the subject of its own article, on retrieval embeddings , and the essential point is that these are not the token embeddings inside a language model but a separate model trained specifically so that a question lands near its answer.\nReranking. The first-stage retrieval is fast and approximate, comparing precomputed vectors. A second stage takes the top fifty or hundred candidates and scores each against the query with a much more expensive model that reads both together — which is far more accurate, because it can attend across the two rather than comparing summaries of each. Reranking is the single highest-value addition to a naive pipeline and is frequently omitted.\nGeneration. The surviving passages go into the prompt with an instruction to answer from them.\nThe distribution of blame is the practical lesson. When these systems give bad answers, the cause is retrieval far more often than generation — the right passage was never fetched, or was fetched and ranked twentieth, or was split in half by the chunker. Debugging the prompt when the retrieval is broken is a common and expensive mistake.\nFigure 2. Four stages, and where the blame actually falls.\nThe seam Several problems arise specifically from joining a search system to a language model, and they have no analogue in either alone.\nThe query is not the question. A user asks \u0026ldquo;what about the second one?\u0026rdquo; and there is nothing in that string to search with. Conversational context has to be folded into the query before retrieval, usually by having the model rewrite it into a standalone form first. Related techniques go further: generate several alternative phrasings and retrieve for each, or have the model write a hypothetical answer and retrieve documents similar to that, on the reasoning that an answer resembles other answers more than a question does.\nPosition matters. As the article on long context describes, models attend less reliably to the middle of their input. Ordering retrieved passages by relevance and dumping them in sequence puts the fourth-best passage in the worst position. Placing the strongest material at the beginning and the end measurably improves answers, which is an unsatisfying thing to have to know but is worth knowing.\nRetrieved text can be wrong, and the model may prefer it anyway. If a fetched passage contradicts what the model knows, the outcome depends on how confident the model is in its own parametric knowledge — and models are, on the whole, quite deferential to their context. That is usually the desired behaviour, since the context is the current information. It also means the retrieval system is a channel through which false or adversarial content reaches the answer, which is a genuine security consideration when documents come from anywhere untrusted.\nAttribution is weaker than it looks. Providing sources is one of the main arguments for the whole approach, but a citation attached by the model is a claim about where the information came from, not a verified link. Models cite passages they did not use and omit ones they did. Systems that need real attribution enforce it structurally — constraining generation to quote, or checking each sentence against the sources afterwards — rather than trusting the model to report honestly.\nWhen to retrieve at all The naive pipeline retrieves on every query, which is wrong in both directions: it wastes context on questions the model can answer unaided, and it retrieves once when a multi-step question needs several rounds.\nThe refinement is to make retrieval a decision rather than a reflex. A model can be trained or prompted to emit a search when it needs one, judge whether the results are sufficient, and search again with a refined query if not. This turns retrieval into tool use — the search becomes one action among several available to a model that is deciding what to do next — and it is the direction essentially all serious systems have moved in. It handles the multi-hop case naturally, since finding the answer to the first part supplies the query for the second, which a single-shot pipeline cannot do at all.\nFigure 3. Retrieval as a reflex, and retrieval as a decision.\nAgainst the alternatives Two comparisons come up constantly and both deserve a straight answer.\nVersus a long context. Now that models accept hundreds of thousands of tokens, why not skip retrieval and supply everything? Sometimes you should. But the corpus is usually far larger than any window; the cache cost of a full context is paid on every request while an index is built once; and the dilution effect means a model given a hundred relevant passages and ten thousand irrelevant ones does worse than one given the ten that matter. Retrieval is a filter, and filtering remains valuable even when the container is large. The techniques also compose — retrieve broadly, then let a long context hold everything retrieved.\nVersus fine-tuning. These address different problems and are constantly confused. Fine-tuning is for teaching a model a behaviour: a format, a tone, a domain\u0026rsquo;s idiom, a task structure. Retrieval is for supplying facts. Attempting to teach facts by fine-tuning works poorly — the model learns them unreliably, cannot be corrected without another round, and gains no ability to cite. Attempting to teach a format by retrieval works poorly for the mirror reason. The right combination for a specialized application is usually both: adapt the model for how to behave, retrieve for what is true.\nWhat to carry forward Retrieval-augmented generation separates a system\u0026rsquo;s knowledge into two kinds — parametric, held in the weights, which is fast and generalizes and cannot be inspected or updated, and non-parametric, placed in the context, which is current and citable and confined to what was fetched — and lets each do what it is good at. The pipeline chunks documents, indexes them as vectors, retrieves the nearest to a query, reranks the candidates with a more expensive model that reads query and passage together, and generates from the survivors; of these, chunking and reranking are the steps most often done badly, and the overwhelming majority of bad answers come from retrieval failures rather than generation failures. The seam between the two systems creates its own problems: a conversational query is not a searchable one and must be rewritten; models attend least well to the middle of a long input, so passage ordering matters; retrieved text overrides the model\u0026rsquo;s own knowledge, which is usually right and makes the retriever a channel for anything false that gets indexed; and citations produced by a model are claims rather than verified links, so real attribution has to be enforced structurally. The naive pipeline retrieves once per query, which is wrong for questions needing several hops and wasteful for those needing none — so serious systems make retrieval a decision the model takes, which turns it into a form of tool use. Against a long context it remains valuable as a filter, since irrelevant material dilutes attention and an index is built once while a context is paid for every request; against fine-tuning it is simply orthogonal, one supplying facts and the other behaviour.\nThe figures in this article follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/articles/rag/","summary":"\u003cp\u003eA model\u0026rsquo;s knowledge is fixed at the end of training, distributed across billions of weights in a form nobody can inspect, edit, or attribute. It cannot tell you where it learned something. It cannot be told that a fact has changed without retraining. And when it does not know an answer it will frequently produce one anyway, because nothing in its training taught it to distinguish confident recall from confident invention. Retrieval-augmented generation addresses all of this by an intervention that is almost embarrassingly direct: before asking the model to answer, go and find the relevant documents, and put them in the prompt. Nearly all the difficulty, however, lives in the retrieval step and in the seam between the two systems, and the naive version fails in ways that are entirely predictable once that seam is understood. This piece is about where the pipeline actually breaks, what the seam demands of both halves, and when retrieval is the wrong instrument to reach for.\u003c/p\u003e","title":"RAG — Giving a Model Something to Read Before It Answers"},{"content":"An ordinary language model has a curious habit: it answers instantly. Give it a question and it produces the next word, then the next, marching straight from the prompt to a reply with no visible pause for thought. For a great many tasks this is fine — recalling a fact, rephrasing a sentence, drafting an email. But ask an ordinary model to multiply two large numbers, or to work through a tricky logic puzzle, or to debug a subtle piece of code, and it often stumbles, blurting out a confident answer that happens to be wrong. It behaves like a student who has been forbidden from using scratch paper and must announce the answer the instant the question is read aloud. Reasoning models are what appears when that ban is lifted. They are trained to think first — to spend time working a problem through, step by step, before committing to an answer. This piece is about what \u0026ldquo;thinking\u0026rdquo; means for a machine that only knows how to predict words, how such models are made, and what the extra room does not buy.\nThe problem with answering instantly To see why instant answers are fragile, remember what a model actually does: at each step it predicts the single next word, given everything so far. All of its reasoning, such as it is, has to happen inside that one forward sweep through the tower of layers — a fixed, finite amount of computation, the same whether the question is \u0026ldquo;what is the capital of France\u0026rdquo; or \u0026ldquo;what is the fifteenth prime number times the seventh.\u0026rdquo; Easy questions and crushingly hard ones are granted exactly the same budget of thought. For a hard problem, that budget simply runs out. The model has no room to try an approach, notice it is going wrong, and back up; it must compress the entire solution into a single reflexive leap. Small wonder it so often leaps wrong.\nBut here is the opening reasoning models exploit. A model can use its own output as a place to think. If, instead of the final answer immediately, the model first writes out a chain of intermediate steps — \u0026ldquo;there are twelve of these, each worth five, so that is sixty, then subtract the four returned…\u0026rdquo; — then each of those steps becomes part of the input for the steps that follow. The words the model writes become scratch paper it can read back. Suddenly a hard problem is no longer squeezed into one forward pass; it is spread across dozens of them, each building on the last. This simple observation — that a model can think out loud, on the page, and lean on what it has written — is the seed of everything that follows.\nFigure 1. One forward pass, or many.\nThinking out loud: the chain of thought The practice of prompting a model to write out its intermediate steps before answering is called chain-of-thought . Early on it turned out to be summonable by nothing more than a polite instruction — appending \u0026ldquo;let\u0026rsquo;s work through this step by step\u0026rdquo; to a prompt — and accuracy on hard problems would jump, sometimes dramatically. The model already had the capability latent within it; it just needed permission to use its output as workspace.\nWhy does this help so much? Partly because it converts one impossible leap into a sequence of easy hops. Predicting the next small step given the work so far is a task the model is very good at, far better than predicting a distant final answer out of thin air. And partly because the written steps act as a kind of external memory. The model does not have to hold the entire half-finished calculation inside a single pass; it can offload it onto the page and read it back, the way an intermediate total gets jotted on the corner of an envelope. The chain of thought is both the reasoning and the notepad the reasoning is written on.\nA reasoning model takes this from a trick you prompt for and bakes it into the model\u0026rsquo;s very nature. Rather than hoping a clever instruction will elicit step-by-step work, the model is trained so that thinking first is simply what it does — often producing a long private stretch of deliberation, sometimes called its thinking tokens, before it ever begins the answer you see.\nMaking a reasoning model: rewarding the right destination So how do you train a model to think well, rather than merely at length? This is where reasoning models depart most sharply from everything before them, and the departure is worth setting out precisely.\nRecall the ordinary way a model learns: it is shown mountains of text and, at each point, nudged to predict the word that actually came next. This teaches it to imitate — to sound like its training data. But genuine problem-solving cannot be taught purely by imitation, because for a truly hard problem there may be no worked solution in the data to copy, and in any case we do not really care whether the model\u0026rsquo;s reasoning looks like a human\u0026rsquo;s. We care whether it arrives at the right answer. That shift — from rewarding imitation to rewarding correct outcomes — calls for a different kind of training, one drawn from reinforcement learning: instead of being shown the answer, the model attempts the problem in its own way and is rewarded when it succeeds and discourage it when it fails.\nThe load-bearing part is where the reward comes from. For a large and important class of problems, correctness can be checked automatically, with no human in the loop. A math answer can be compared against the known result. A piece of code can be run against a battery of tests. A logic puzzle has a definite solution. This is the idea behind what is often called reinforcement learning from verifiable rewards : let the model generate a full chain of thought and a final answer, mechanically check whether the answer is right, and use that verdict as the training signal. Get it right, and the whole path of reasoning that led there is reinforced — made a little more likely next time. Get it wrong, and that path is discouraged. Repeat across millions of problems.\nWhat makes this so powerful is that the model is never told how to think. It is told only whether it succeeded. Left to search for whatever reasoning earns the reward, models trained this way spontaneously develop striking habits — breaking problems into parts, checking their own work, noticing a mistake mid-stream and saying, in effect, \u0026ldquo;wait, that is wrong, let me reconsider,\u0026rdquo; and backtracking to try another route. Nobody scripted those behaviours. They emerged because they help get the answer right, and getting the answer right is what earns the reward. The model discovers, on its own, that careful deliberation pays.\nThe cost of the right answer, paid at inference Once such a model exists, the second half of the story unfolds not during training but every single time the model is used. And it upends a comfortable old assumption.\nFor most of the field\u0026rsquo;s history, the way to make a model better was to make it bigger or train it longer — to pour more effort into building it, after which running it was a fixed, one-shot expense. Reasoning models add a second lever, pulled at the moment of use: let the model think longer. This is what people mean by test-time compute — spending more computation at the moment the question is posed, not only when the system was trained. Because the model\u0026rsquo;s thinking is written out as tokens, thinking longer literally means generating more of them: a longer chain of deliberation before the answer. And the remarkable, repeatable finding is that on hard problems, accuracy climbs with the room the model is given to think. Grant it a few hundred words of scratch work and it does better; give it room for several thousand and, up to a point, it does better still.\nThis is precisely the scratch-paper effect described above, now turned into a dial that can be set at will. An easy question needs only a moment\u0026rsquo;s thought and it would be wasteful to spend more. A hard one deserves a long, winding deliberation with false starts and self-corrections. The same model can do either; the only choice being made is how much room to think it is granted. There are even more elaborate ways to spend this budget — having the model explore several different lines of reasoning and then favour the answer they most agree on, for instance — but the essential insight is the humble one: a model given more time to think, on the page, tends to think better.\nTwo budgets, and how they trade It is worth holding the two halves of the story side by side, because reasoning models make a genuinely new trade-off available. The first budget is the old one, spent up front: the cost of training the model. The second is spent at the moment of use: the cost of letting it think. What is striking is that these budgets can substitute for each other. A problem a smaller model can crack given room to deliberate might otherwise have demanded a far larger, costlier model answering in one shot. Accuracy can be bought either by training a larger model or by giving a smaller one longer to work, and often the second is cheaper. The accounting is set out under test-time compute , which also separates this purchase from a different one: letting one attempt think longer buys depth, sampling many attempts buys coverage, and the two fail in different places. This is why reasoning models reshaped the field\u0026rsquo;s sense of where progress comes from: not only from ever-larger training runs, but from teaching models to spend effort wisely at the moment it matters.\nThe trade has its limits. Thinking longer costs real time and money at every use, so a model that muses for thousands of words on a trivial question is a poor bargain — hence ongoing work on teaching models to match their effort to the difficulty of the problem, thinking hard only when hard thinking is called for. And more thinking is not infinitely helpful; beyond some length the gains flatten, and a model can even talk itself out of a correct answer. Deliberation is a powerful tool, not a magic one.\nFigure 2. Two budgets that substitute for each other.\nPassing the skill along There is a coda, and it connects to a technique worth knowing on its own. A large, expensive reasoning model that has learned — through all that costly reinforcement — to produce excellent chains of thought can serve as a teacher. It works through a great many problems, the good reasoning traces are collected, and a much smaller model is trained to imitate them. The small model never has to rediscover how to reason through the expensive search; it simply learns to copy the worked-out thinking of a model that already knows how. This is distillation applied to reasoning traces: the hard-won skill of deliberation, transferred from a giant into something lean enough to run cheaply. It is much of the reason capable reasoning is now available in models small enough to run almost anywhere.\nFigure 3. From prompt trick to trained habit to distilled skill.\nWhat to carry forward Ordinary models answer in a single reflexive leap, granting every question the same fixed sliver of thought — which is why they stumble on genuinely hard problems. Reasoning models lift that constraint by letting the model think out loud, writing a chain of intermediate steps that becomes scratch paper it can read back, spreading a hard problem across many passes instead of cramming it into one. They are trained not to imitate human reasoning but to reach correct answers, using reinforcement learning with rewards that can be checked automatically — right answer, reinforce the path; wrong answer, discourage it — from which habits like self-checking and backtracking emerge on their own. And crucially, their quality can be dialled up at the moment of use by letting them think longer, a lever called test-time compute that can substitute for sheer model size. The result is a genuinely new way to spend effort: not only in building the model, but in giving it room to think before it speaks, and through distillation in handing that skill down to models small enough for everyday use. The room is not free and it is not unbounded. Thinking is billed at every use rather than once at training, beyond some length the gains flatten and a model can talk itself out of a correct answer, and the trace that makes the deliberation visible is not therefore a description of it — deliberation is a powerful tool and not a magic one.\nThe figures in this article follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/articles/reasoning-models/","summary":"\u003cp\u003eAn ordinary language model has a curious habit: it answers instantly. Give it a question and it produces the next word, then the next, marching straight from the prompt to a reply with no visible pause for thought. For a great many tasks this is fine — recalling a fact, rephrasing a sentence, drafting an email. But ask an ordinary model to multiply two large numbers, or to work through a tricky logic puzzle, or to debug a subtle piece of code, and it often stumbles, blurting out a confident answer that happens to be wrong. It behaves like a student who has been forbidden from using scratch paper and must announce the answer the instant the question is read aloud. \u003cem\u003eReasoning models\u003c/em\u003e are what appears when that ban is lifted. They are trained to think first — to spend time working a problem through, step by step, before committing to an answer. This piece is about what \u0026ldquo;thinking\u0026rdquo; means for a machine that only knows how to predict words, how such models are made, and what the extra room does not buy.\u003c/p\u003e","title":"Reasoning Models — Teaching a Model to Think Before It Speaks"},{"content":"Anyone who learned this subject from the textbooks of the twenty-tens learned that the central hazard of fitting a flexible model to data was overfitting, and learned an apparatus for holding it off: a penalty on the size of the weights, noise injected into the activations, a held-out split watched for the moment its curve turns up, augmentation to manufacture examples the collection lacked. That training was sound and the techniques worked. Then you read a technical account of a large language model and find almost none of it. No noise injection. No stopping criterion based on held-out loss, because nothing turns upward to stop at. A weight decay coefficient, retained, listed among the settings with no explanation beyond its value. The tempting conclusion — that the received wisdom was wrong, or that scale makes carelessness affordable — is not what happened. This piece is about which precondition stopped holding, what the surviving weight decay is actually doing, and what took over the job the old defences performed.\nThe received picture was correct about a setting that no longer obtains Begin with the classical account, because it describes something real. Take a collection of some tens of thousands of examples and a model with far more capacity than the collection can pin down, and train it as long as you like. The training loss falls and keeps falling, because there is always some further contortion that fits the examples in front of it a little better. The held-out loss falls too, for a while, then bottoms out and rises. Past the turning point the model is learning the noise in the sample rather than the structure behind it.\nThe apparatus exists to manage that curve, and each piece attacks it differently. A penalty on the norm of the weights makes the contortions expensive, so the fit has to earn its complexity. Injected noise stops any unit from relying on the precise behaviour of its neighbours, since the neighbour may not be there. Augmentation enlarges the sample with transformations that ought not to change the answer. And a held-out split supplies what all of it needs: a signal for where the turning point is, which the training loss will never give.\nNone of this was mistaken and none of it has been refuted. What is easy to miss is that the argument was conditional. It assumed a model with more capacity than the data could constrain, and enough passes over that data for the model to start fitting its idiosyncrasies. Both conditions were so dependable that nobody stated them. One has stopped being true.\nOne pass over a corpus larger than the model changes the binding constraint A large language model is trained for roughly a single pass over a corpus containing far more text than it has parameters to store. Most training examples are seen once and never again. There is no second epoch in which the model can memorize what it saw in the first, because for the bulk of the data there is no second epoch at all.\nSo the classical curve is not what practitioners observe. The held-out loss does not turn upward. It goes down, more and more slowly, in the regular way that scaling laws describe, until the compute budget runs out and the run stops. The binding constraint has moved. In the old setting the scarce resource was data and the model was large relative to it; now the data is effectively inexhaustible and the scarce resource is arithmetic. Scaling laws are the statement of that second regime, which is why they say nothing about when to stop and everything about how to spend.\nThis is where dropout went. It was standard in the models of the twenty-tens and is largely absent from large pretraining runs, and the reason is mechanical rather than fashionable. Dropout injects noise that prevents units from co-adapting on data they will meet again, and it charges for the service: the network trains as a blurred ensemble of subnetworks, giving up effective capacity, and converges more slowly because each step is taken on a degraded model. In a data-limited setting that is an excellent trade. In a single-pass, compute-limited setting it pays a real price in the scarce resource to suppress a failure that is not occurring. Dropout has not left the field, though, only pretraining. It survives wherever the old preconditions hold — fine-tuning on a few thousand examples for many epochs, with a model vastly larger than the task needs, which is the classical picture exactly.\nFigure 1. The curve that turns, and the curve that does not.\nWeight decay survived, and it is not doing what its name suggests Weight decay did not follow dropout out of the door, which is odd if its purpose was capacity control, since the argument for dropping dropout applies word for word. Something else is going on.\nTwo clues. The first is that in the optimizer these models are trained with, the decay is not a term in the loss; it is a separate shrinkage applied to every weight after the gradient step, kept deliberately outside the adaptive rescaling. Any intuition inherited from treating it as a prior is already strained by that surgery. The second is that removing it does not produce the signature of failed regularization. Runs without decay do not fit their training data suspiciously well; they optimize worse.\nThe reading that fits the evidence is that decay controls the scale of the weights, and the scale of the weights controls the effective size of a step. A learning rate is a fixed absolute quantity, so as a weight matrix grows in norm the same step becomes an ever-smaller relative change to what the layer computes, and the model gradually stops moving while appearing to train normally. Decay puts a ceiling on that growth and keeps the relative step size roughly stationary over a long run. On this account it is not preventing memorization; it is maintaining the conditions the optimizer needs to keep making progress, which places it nearer to a normalization trick than to a penalty. Accounts of the mechanism differ and the argument is unsettled. The empirical fact is not: the coefficient earns its place in every configuration file, and the explanation printed beside it is not the reason.\nA pinch of math: two objectives, and a dial that moves when you move another Put the two readings side by side; the difference is one line of algebra.\nThe classical form adds a penalty to the objective:\n$$L(\\theta) + \\frac{\\lambda}{2}\\lVert\\theta\\rVert^2$$and the reading follows from the shape. Any setting of the parameters must justify its norm by the loss it buys, so large-norm solutions are excluded unless they fit substantially better — the hypothesis space is shrunk, and $\\lambda$ says by how much.\nThe decoupled form does not touch the objective. It shrinks the weights directly, alongside whatever step the optimizer computed from the gradient estimate $\\hat{g}$:\n$$\\theta \\leftarrow \\theta - \\eta\\,\\hat{g} - \\eta\\lambda\\theta$$Now look at the last term. The shrinkage is multiplied by the learning rate $\\eta$, so each step retains a fraction $(1 - \\eta\\lambda)$ of every weight — a fraction set by a quantity belonging to the gradient half of the update. Decay the learning rate, as every schedule does, and you have weakened the regularization by the same factor, at exactly the point where the classical story says restraint matters most. The penalized objective predicts nothing of the kind: there, $\\lambda$ describes the problem and $\\eta$ describes only how fast we solve it. Here the two are welded, and the welding follows from the update rule rather than from any argument about generalization. Decoupling bought independence from the per-parameter gradient scale and left this entanglement standing. So the effective strength of the decay is a product of two dials, not the number in the configuration.\nFigure 2. Effective decay is a product of two dials.\nWhat took over the job, and what it quietly fails to do If the old defences are gone and the loss still behaves, something is doing the work. Three things are, and none is a term in the objective.\nDeduplicating the corpus does the most. The web repeats itself relentlessly, and a document appearing forty times in the raw crawl is forty opportunities to spend capacity on a specific string rather than on the language around it. Removing them removes the only part of a single-pass regime that behaves like multiple epochs. Notice what kind of intervention that is: it fixes the data rather than constraining the model, and it belongs to the mixture pipeline, not the optimizer.\nThe noise in the gradients regularizes for free. Each step is taken on an estimate assembled from a sample, and the resulting jitter keeps the parameters from settling into narrow crevices of the landscape. Nobody adds it and nobody pays for it. But it shrinks as the batch grows, so a very large batch removes a stabilizer along with the variance — one of several reasons the largest batches disappoint at equal data.\nAnd the corpus itself is the real defence. A model cannot memorize its way to a low loss on a trillion tokens it will see once, because the capacity is not there and the second look is not coming. That is the least elegant answer available and the load-bearing one.\nNone of that prevents memorization. The absence of the classical curve is easily misread as evidence that nothing is being stored. Verbatim retention of rare sequences happens, it is measurable, and it is a different phenomenon from the aggregate loss, which is an average over billions of tokens and insensitive to a few thousand of them. The reassuring curve and the memorization problem coexist comfortably.\nAnd nothing tells you when to stop. Early stopping was a decision made on evidence: watch the held-out curve, halt at the minimum. Where the curve never turns, that evidence does not exist. The endpoint is fixed instead by the budget and written into a schedule that must reach zero on time, so the length of a run is chosen before the run has said anything about itself. What was a measurement is now a procurement decision.\nFigure 3. What replaced the old defences.\nWhat to carry forward The classical account of overfitting was correct about a setting defined by two conditions so dependable that nobody wrote them down: a model with more capacity than the data could constrain, and enough passes over that data for the model to begin fitting the sample rather than the structure. Large-scale pretraining satisfies neither. Roughly one pass over a corpus far larger than the model means most examples are seen once and there is no second epoch in which to memorize the first, so the held-out loss does not turn upward — it falls until the arithmetic runs out, which is what makes the regime compute-limited rather than data-limited and what the scaling laws describe. Dropout left because it charges effective capacity and slower convergence to suppress co-adaptation on repeated data, and there is no repeated data; it remains sensible in fine-tuning on small collections, where the old conditions hold exactly. Weight decay stayed, and its name has outlived its mechanism: applied as direct shrinkage outside the loss, its benefit looks less like capacity control than like keeping the norm of the weights bounded so that a fixed learning rate remains a meaningful relative change — an explanation still contested, over an effect that is not. Its strength is multiplied by the learning rate, so every decay of the schedule quietly weakens it. What performs the old function now is deduplication, gradient noise that thins as batches grow, and the sheer size of the corpus, the least elegant of the three and the one carrying the weight. None of them guarantees anything against verbatim retention, which happens without disturbing the loss curve, or supplies a signal for when to stop, which is now set by a budget fixed before the evidence exists.\nThe figures in this article follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/articles/regularization/","summary":"\u003cp\u003eAnyone who learned this subject from the textbooks of the twenty-tens learned that the central hazard of fitting a flexible model to data was overfitting, and learned an apparatus for holding it off: a penalty on the size of the weights, noise injected into the activations, a held-out split watched for the moment its curve turns up, augmentation to manufacture examples the collection lacked. That training was sound and the techniques worked. Then you read a technical account of a large language model and find almost none of it. No noise injection. No stopping criterion based on held-out loss, because nothing turns upward to stop at. A weight decay coefficient, retained, listed among the settings with no explanation beyond its value. The tempting conclusion — that the received wisdom was wrong, or that scale makes carelessness affordable — is not what happened. This piece is about which precondition stopped holding, what the surviving weight decay is actually doing, and what took over the job the old defences performed.\u003c/p\u003e","title":"Regularization at Scale — Why the Old Defences Stopped Mattering"},{"content":"The article on embeddings concerned the table at the bottom of a language model, which turns each token into a vector so the tower has something to work with. The object here is a different one with an unfortunately identical name: a model whose entire output is a single vector representing a whole passage, built so that passages about the same thing land near each other and can be found by searching. The two share a word and a geometric intuition and almost nothing else. The distinction is worth insisting on, because a great deal of confusion in practice comes from expecting one to do the other\u0026rsquo;s job — and in particular from the reasonable-seeming and wrong idea that a good language model must contain a good search index. This piece is about why it does not, what the training objective actually asks for, and what the resulting vectors cannot be made to do.\nWhy a language model is not one Suppose you want to find, among a million passages, the one that answers a question. The obvious thought is to use the model you already have: run each passage through it and take some summary of the internal state as its representation.\nThis works badly, and the reason is instructive. A language model is trained to predict the next token. Its internal state at any position is optimized to be useful for that prediction, which means it is dominated by local, syntactic, immediately-relevant information — what part of speech is coming, which entity is currently in play, whether a clause is closing. It is not optimized to summarize a passage in a way that supports comparison with other passages. Nothing in the training objective ever asked for that.\nThere is a more specific problem. Averaging a model\u0026rsquo;s token representations produces vectors that are all rather similar to one another, because they share the dominant directions that every position carries — the anisotropy noted in the article on embeddings , where learned representations occupy a narrow cone rather than spreading out. Similarity scores computed this way are all high and differ in the third decimal place, which is a poor foundation for ranking a million candidates.\nA retrieval model is therefore trained separately, for a different objective, and usually initialized from an encoder -style architecture that reads bidirectionally rather than left-to-right — because a passage being summarized is available all at once, and there is no reason to hide its second half from its first. The construction that follows is the same one that, applied across modalities rather than within one, produces the vision encoders of multimodal models .\nWhat the training objective actually is The requirement is easy to state: a question and its answer should be close; a question and an unrelated passage should be far. The training method that achieves it is contrastive, and its central quantity is not similarity but relative similarity.\nTake a batch of question-passage pairs known to belong together. For each question, its own passage is the positive and every other question\u0026rsquo;s passage in the batch is a negative. Then apply a softmax over the similarities and maximize the probability of the correct one:\n$$\\mathcal{L} = -\\log \\frac{e^{\\,\\text{sim}(q, p^{+})/\\tau}}{\\sum_{j} e^{\\,\\text{sim}(q, p_j)/\\tau}}$$This is cross-entropy over a set of candidates rather than over a vocabulary, and reading it that way makes the design decisions legible.\nThe objective is about ranking, not about absolute distance. Nothing requires the correct pair to reach any particular similarity; it only has to beat the alternatives. This is why the scores such a model produces are not calibrated and should not be thresholded as if they were probabilities.\nThe temperature $\\tau$ does what it does everywhere: a small value sharpens the distribution and concentrates the gradient on the hardest negative, a large one spreads it. It matters more here than in most places, and is typically small.\nAnd the negatives are the whole game. Random passages are trivially distinguishable — a question about tax law versus a passage about volcanoes teaches almost nothing after the first few steps. What produces a good model is hard negatives: passages that look relevant and are not. The standard method is to retrieve candidates with a current version of the model, discard the true positives, and train against the highest-ranked remainder, iterating as the model improves. Since negatives are shared across the batch at almost no extra cost, larger batches supply more of them, which is why these models are trained with unusually large batches and with mechanisms for holding many more negatives in memory than the batch itself contains.\nThe other lever is symmetry. Questions and passages look nothing alike — one is short and interrogative, the other long and declarative — so many systems use two separate encoders, one per side. Others use a single model with a short instruction prefixed to each input telling it which role it is playing, which is simpler to serve and works about as well.\nFigure 1. Hard negatives are where the learning is.\nBi-encoders and cross-encoders There are two ways to score a query against a passage, and the difference between them determines the shape of every retrieval system.\nA bi-encoder encodes each independently and compares the resulting vectors. The passage vectors can be computed once, in advance, for the entire corpus. At query time only the question must be encoded, and the search is a nearest-neighbour lookup among precomputed points. This scales to billions of documents.\nA cross-encoder takes query and passage together as one input and outputs a relevance score. Attention runs across both, so every word of the question can interact with every word of the passage — which is far more accurate, because relevance often depends on a specific correspondence that no independent summary of either side can preserve. It also means nothing can be precomputed: every query-passage pair requires a full forward pass, so scoring a million passages requires a million forward passes.\nThe resolution is the two-stage arrangement described under RAG . A bi-encoder retrieves a few hundred candidates cheaply from the whole corpus; a cross-encoder reranks those few hundred expensively. Cost scales with the corpus for the cheap stage and with the shortlist for the accurate one. Between the two sits a middle option that keeps a vector per token rather than per passage and computes a fine-grained match at query time — more accurate than a single vector, far cheaper than full cross-attention, and considerably more storage.\nFigure 2. Bi-encoder scales, cross-encoder discriminates.\nWhat the vectors cannot do Three limitations are worth knowing because they explain most disappointing results.\nExact matching is a weakness, not a strength. A dense vector represents a passage approximately, which is what lets it match paraphrases — and it is exactly wrong for finding a specific product code, an unusual surname, an error string, or a rare technical term. Traditional keyword search handles those perfectly and handles paraphrase not at all. The two are complementary in the strict sense, which is why serious systems run both and merge the rankings, an arrangement usually called hybrid search. Reaching for dense retrieval alone because it is newer is a common and easily avoided mistake.\nA single vector is a bottleneck. A long passage covering several topics gets one point in space, which is an average of its contents and therefore near nothing in particular. This is really an argument about chunking: passages should be short enough that one vector honestly represents them.\nRelevance is not one thing. The model has learned some particular notion of relatedness from its training pairs, and a query may want a different one — the most recent document, the most authoritative, the one from the right jurisdiction, the one that contradicts rather than matches. None of these are geometric properties of a semantic embedding, and expecting a vector search to honour them leads to protracted confusion. They belong in metadata filters and in ranking logic outside the vector.\nFigure 3. Dense matches paraphrase; keyword matches the identifier.\nPractical matters Two brief points that come up immediately in use.\nSearching a billion vectors exactly is too slow, so approximate methods are used — layered proximity graphs are the common choice — which trade a small chance of missing a true nearest neighbour for enormous speed. The recall loss is usually negligible next to the errors the embedding model itself makes.\nAnd storage is real: a billion passages at a thousand dimensions in four bytes is four terabytes. The vectors compress well, since retrieval needs only relative ordering rather than precise values, and the same quantization logic applies. Some models are now trained so that a prefix of the vector is usable on its own, which allows searching cheaply on a truncated version and refining with the full one — the same coarse-then-fine pattern as the two-stage pipeline, applied within a single vector.\nWhat to carry forward A retrieval embedding is a separate model from the token embedding table it shares a name with: it maps a whole passage to one vector chosen so that related passages land near each other. A language model\u0026rsquo;s internal states do not serve this purpose, because they are optimized to predict the next token rather than to summarize a passage for comparison, and because they occupy a narrow cone that makes all similarities look alike. The training is contrastive — a softmax over candidates in which a question\u0026rsquo;s own passage must outrank every other, which makes the objective about relative ranking rather than absolute distance, so the scores are not calibrated and should not be thresholded as probabilities. Its quality is determined largely by the negatives: random passages teach nothing after the first steps, so systems mine hard negatives by retrieving with the current model and training against the top-ranked wrong answers, iterating, and they use very large batches because negatives are shared across a batch for free. Scoring comes in two forms — a bi-encoder that embeds each side independently, allowing the corpus to be precomputed and searched at scale, and a cross-encoder that reads query and passage together for far better accuracy at a cost that scales with the number of pairs — which is why the standard pipeline retrieves broadly with the first and reranks narrowly with the second. The limitations follow from what a dense vector is: it is approximate, so it matches paraphrases well and exact identifiers badly, which is why hybrid search with keyword matching is standard; one vector per passage is a bottleneck that argues for short chunks; and notions of relevance like recency, authority, or contradiction are not geometric and belong outside the vector entirely.\nThe figures in this article follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/articles/retrieval-embeddings/","summary":"\u003cp\u003eThe article on \u003ca href=\"/articles/embeddings/\"\u003eembeddings\u003c/a\u003e\n concerned the table at the bottom of a language model, which turns each token into a vector so the tower has something to work with. The object here is a different one with an unfortunately identical name: a model whose entire output is a single vector representing a whole passage, built so that passages about the same thing land near each other and can be found by searching. The two share a word and a geometric intuition and almost nothing else. The distinction is worth insisting on, because a great deal of confusion in practice comes from expecting one to do the other\u0026rsquo;s job — and in particular from the reasonable-seeming and wrong idea that a good language model must contain a good search index. This piece is about why it does not, what the training objective actually asks for, and what the resulting vectors cannot be made to do.\u003c/p\u003e","title":"Retrieval Embeddings — Turning Meaning Into a Searchable Coordinate"},{"content":"Training a language model is, up to a point, mechanical: choose a loss, differentiate it, descend. Each stage works because it has something concrete to compare against — the next token in the corpus, a demonstration written by a person, the verdict of a test suite. Then the requirement becomes be helpful, and the machinery has nothing to bite on. Nobody can write a differentiable expression whose minimum is a tactful reply, or one that prefers the right amount of hedging to slightly too much. The properties are real; they have no closed form. What people can do is look at two answers to the same question and say which is better, often without being able to say why. Preference-based alignment rests on one substitution made at that point: fit a function to the comparisons, then treat the function as the objective. This piece is about how that function is built, what the substitution buys, and the ways it goes wrong.\nRatings drift, comparisons hold still The obvious way to gather data about quality is to ask for it directly: show an annotator a response, ask for a score out of ten, fit a regressor to a hundred thousand of them. It works badly, for reasons that have nothing to do with the model. A seven means different things to different people, because each annotator is privately inventing the scale as they use it. Worse, the scale moves. Two hours of mediocre answers recalibrates a rater without their noticing, so a response that earned a six in the first hour earns an eight in the third, and the drift leaves no trace.\nA comparison asks for much less. The annotator never has to locate an answer on an absolute scale, only to break a tie against a reference sitting on the screen. That judgement is markedly more stable, between people and within a long session, because most of what corrupts an absolute rating is a property of the rater\u0026rsquo;s state — and it corrupts both members of a pair identically, so it cancels.\nThe cost is information. A comparison carries close to one bit where a rating carried a number, so far more are needed, and it records order without distance: a marginally cleaner answer and an unusable one produce the identical annotation. And a coin flip on an indistinguishable pair enters the data looking exactly like a preference.\nThe mechanism is the model with its head replaced The construction is unromantic. Take a language model, discard the unembedding matrix that would have projected the final hidden state onto the vocabulary, and bolt on a single linear head producing one number. Run the prompt and the whole response through it, read the scalar at the last position, and that is the reward. The same weights that were reading text to predict tokens are now reading it to produce a score.\nInitialization matters more than it looks. The head is attached to the supervised fine-tuned model rather than to a base model, because a reward model started from scratch would have to learn what English is before it could learn what a good answer is, and a hundred thousand preference pairs cannot teach both. The head reads off a distinction the model can already draw.\nTraining is a logistic loss over the pairs, run for very few passes, since preference data is scarce and these models overfit within an epoch or two. And the scale is arbitrary. Add a constant to every score and the loss is unchanged; nothing pins down where zero sits. Only differences mean anything. It is the situation of a logit , meaningless in isolation while a difference of two logits is an honest log-odds — so a reward cannot be compared across models or runs, or read as a grade.\nA pinch of math: the logistic model of a preference To fit a function to comparisons we need a rule turning two scores into the probability a person picks one. The standard choice is the Bradley–Terry model, borrowed from the statistics of paired comparisons long before anyone applied it to text. For a prompt $x$, a preferred response $y_w$ and a rejected one $y_l$, it posits:\n$$P(y_w \\succ y_l \\mid x) = \\sigma\\big(r(x, y_w) - r(x, y_l)\\big), \\qquad \\sigma(z) = \\frac{1}{1 + e^{-z}}$$Each piece is there for a reason. The subtraction is there because a preference is a comparison and nothing else was ever observed. The logistic $\\sigma$ squashes an unbounded real difference into a probability, and behaves sensibly at the boundaries: equal scores give one half, a large gap near-certainty, and the transition between is differentiable. Maximum likelihood over such triples gives the loss the apparatus is trained with,\n$$\\mathcal{L} = -\\log \\sigma\\big(r_\\theta(x, y_w) - r_\\theta(x, y_l)\\big)$$a two-outcome cross-entropy, structurally identical to what any binary classifier minimizes. Two things fall out of it.\nThe reward is identified only up to an additive constant per prompt. Replace $r(x,y)$ by $r(x,y) + c(x)$ for any function $c$ of the prompt alone, and every within-prompt difference is untouched, so the loss sits where it was. The model never learns an absolute standard: its scores compare two answers to one question and nothing across questions, which is why every downstream use supplies its own reference point, and why policy optimization leans on baselines and advantage whitening.\nAnd the gradient separates a pair by as much as it is currently wrong. Writing $\\Delta = r_w - r_l$, the derivative of the loss with respect to the margin is $-\\big(1 - \\sigma(\\Delta)\\big)$, so the scores are pushed apart in proportion to the probability the model assigns to the wrong ordering. A pair already ranked correctly by a comfortable margin contributes almost nothing; a pair it gets confidently backwards dominates. Training concentrates on its own mistakes, and nothing bounds the margin: where every annotator agrees, the loss keeps paying for a wider gap forever. That is why the run is kept short.\nFigure 1. Bradley–Terry, and where the gradient goes.\nFigure 2. Only differences within a prompt mean anything.\nWhat the substitution costs Three costs follow from fitting a function to preferences and then optimizing it, and they differ in kind.\nIt is a proxy, and proxies get gamed. The reward model learned correlations from a finite pile of comparisons, some tracking quality, some accidents of collection. Longer answers scored slightly better on average, so length becomes a lever. Bulleted structure looked thorough, so everything arrives in bullets. Agreement scored better than contradiction, so the model becomes agreeable — sycophancy is not a character flaw picked up somewhere but a correlation in the labels, exploited as instructed. Confident phrasing outscored hedged phrasing, so hedging thins out whether or not there are grounds. None of this is misbehaviour. It is maximizing the number it was given.\nIt is only valid near the data it was trained on. The fitted function is accurate where comparisons were collected and unconstrained everywhere else. As the policy improves it leaves that region for territory the reward model has never scored, where a smooth surface pushed hard tends to have spurious peaks. So the reward becomes least trustworthy at the moment training is going well. This is why a KL penalty to the reference model exists at all: not a refinement but a fence against travelling to where its judge cannot see. The fence also caps how much can be learned, which is the bind the method never escapes.\nAgreement between annotators is itself mediocre. On subjective quality, two people shown the same pair often agree around seventy percent of the time, and that is a ceiling rather than an obstacle to be engineered away: no model can predict a judgement the judges do not share. A reported accuracy in the low seventies is therefore at the noise floor, not a result. Worse, the biases in human raters — toward length, confidence, and fluency over accuracy — are the same ones described above as reward hacking, which the article on evaluation reaches from the other side. The hacking is not a distortion introduced by the fitting. It is faithful transmission of a distortion already in the source.\nFigure 3. Rater bias in, reward hacking out.\nWhat it cannot say, and the two escapes Beyond the costs, there are two things a reward model does not do at all, and neither is a matter of accuracy.\nIt scores a whole response with one number. A five-hundred-token answer with one confused paragraph and four good ones receives a single scalar, and nothing in it says which paragraph was the problem. The reinforcement learning downstream spreads that verdict across every token, so a mostly-good response is punished uniformly and a mostly-bad one rewarded uniformly, leaving the model to infer where the error was from many attempts. Process-level scoring, grading each step of a derivation, exists to address this, and it needs a labeller for those steps — people, which is expensive, or a model, which reintroduces the proxy.\nIt cannot represent disagreement. Where people genuinely differ — how blunt an answer should be, how much a difficult topic should be softened, where a refusal belongs — the fitting does not record a split. It averages. The result is a target between two incompatible positions, corresponding to nobody\u0026rsquo;s actual preference, and a policy optimized against it converges on a compromise no annotator would have chosen. The honest object would be a distribution over rewards, or several models for several populations. A single scalar has no room to carry the disagreement, so it is quietly deleted.\nTwo escapes exist, with limits worth stating. The first removes the model: where an answer can be checked by a program, verifiable rewards substitute a test suite or a proof checker for the learned judgement, and the reward becomes a fact that cannot be flattered. The second removes the separate training stage: direct preference optimization reads the reward off the policy itself as a log-ratio against its starting point, deleting the network without deleting the reliance on preferences. Neither reaches the territory that motivated the construction. For everything no program can check, comparisons are the only instrument available, and a function fitted to them the only objective on offer.\nWhat to carry forward The reward model exists because there is no loss function for a good answer: helpfulness, tone, tact and appropriate hedging are properties with no closed form, while a person shown two responses can say which is better without being able to say why. Every problem descends from the substitution: fit a function to the comparisons, then optimize it. Comparisons rather than absolute scores, because ratings are badly calibrated between annotators and drift within a session, whereas what corrupts a rating usually corrupts both members of a pair and cancels in the choice; the price is a label carrying roughly one bit, recording order without distance. The construction is the language model with its unembedding replaced by a scalar head, initialized from the supervised fine-tuned model, and trained briefly with a logistic loss under the Bradley–Terry assumption that a preference is a logistic function of a difference of rewards. Two properties follow: the reward is fixed only up to an additive constant per prompt, so scores compare within a question but not across questions, and every downstream use supplies its own baseline; and the gradient separates the two scores in proportion to how confidently the current ordering is wrong, with nothing bounding the margin, which is why the run is kept short. The costs are that a proxy gets gamed — length inflation, formulaic structure, sycophancy, confident phrasing displacing warranted uncertainty; that its validity is local, so the reward degrades as the policy improves and moves off the data, which is why the KL fence is load-bearing; and that annotator agreement near seventy percent caps achievable accuracy, making a reported figure at that level the noise floor, not a result. What it never does is say which part of a response was bad, or admit that people disagree.\nThe figures in this article follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/articles/reward-models/","summary":"\u003cp\u003eTraining a language model is, up to a point, mechanical: choose a loss, differentiate it, descend. Each stage works because it has something concrete to compare against — the next token in the corpus, a demonstration written by a person, the verdict of a test suite. Then the requirement becomes \u003cem\u003ebe helpful\u003c/em\u003e, and the machinery has nothing to bite on. Nobody can write a differentiable expression whose minimum is a tactful reply, or one that prefers the right amount of hedging to slightly too much. The properties are real; they have no closed form. What people can do is look at two answers to the same question and say which is better, often without being able to say why. Preference-based alignment rests on one substitution made at that point: fit a function to the comparisons, then treat the function as the objective. This piece is about how that function is built, what the substitution buys, and the ways it goes wrong.\u003c/p\u003e","title":"Reward Models — Turning Human Taste Into a Number"},{"content":"Almost every model in wide use has learned by being shown what a good answer looks like. In pretraining the good answer is simply the next token in the corpus. In supervised fine-tuning it is a demonstration written by a person. In preference learning it is whichever of two responses a human preferred — or, more precisely, whichever a reward model, itself trained to imitate those humans, predicts a human would have preferred. That last substitution is a compromise the field has quietly lived with for years: the thing doing the grading is a neural network with opinions and blind spots of its own. Reinforcement learning from verifiable rewards, usually shortened to RLVR, is what becomes possible when you can throw the grader away and simply check the answer. This piece is about what changes when the reward stops being a judgement and starts being a fact.\nThe problem with learning from a judge you also trained Preference learning works roughly like this. Collect pairs of model responses, ask people which they prefer, train a smaller model to predict those preferences, and then use that model\u0026rsquo;s score as a reward while the main model generates. The reward model is a stand-in for human taste, and it has to be, because you cannot put a person in the loop for the millions of judgements that reinforcement learning consumes.\nThe trouble is that a stand-in can be satisfied without the original being satisfied. The reward model has learned correlations from a finite set of comparisons, and some of those correlations are real signals about quality while others are accidents of how the data was gathered. Longer answers scored slightly better on average, so length becomes a lever. Confident phrasing scored better than hedged phrasing, so hedging disappears whether or not the model has grounds for confidence. Bulleted structure looked thorough, so everything arrives in bullets. None of this is deception on the model\u0026rsquo;s part; it is doing exactly what we asked, which is to maximize a number. It is our number that was wrong.\nThis is reward hacking, and it is not an occasional malfunction but the expected outcome of pushing hard on an imperfect proxy. Which is precisely why the standard recipe includes a leash: a penalty on how far the model\u0026rsquo;s output distribution is allowed to drift from where it started, so that it cannot wander into the strange regions of language where the reward model\u0026rsquo;s scores are highest and its judgement is worst. The leash keeps things safe, and it also caps how much can be learned. You are permitted only small departures from the model you began with, because large departures are exactly where the proxy stops being trustworthy.\nSo the ceiling on preference learning is not really about the algorithm. It is that we do not have a reward we can push on indefinitely.\nA reward you cannot flatter For some questions, we do. The answer to a competition mathematics problem is a specific number, and a short program can compare the model\u0026rsquo;s final answer against it. A function is supposed to sort a list, and you can run it against a hundred test cases. A proof written in Lean either typechecks or it does not. A generated SQL query either returns the expected rows or it returns something else. A puzzle either satisfies its constraints or it violates one.\nIn each of these the reward is not an estimate of quality. It is a fact about the world, produced by a verifier — a piece of ordinary software, deterministic, cheap to run, with no parameters and no opinions. A unit test cannot be charmed by a confident tone. A theorem prover does not find bulleted structure persuasive. The lock either opens or it does not.\nThis changes the economics of the whole procedure. Because the reward is honest, you can optimize against it far more aggressively than you would ever dare against a learned one. The leash can be loosened or, in some setups, removed entirely, and the model is free to travel a long way from where it started. It can develop habits that no human demonstration contained, and as long as those habits raise the fraction of locks that open, they will be reinforced. The one bit of information — right or wrong — is thin, but it is incorruptible, and it turns out that incorruptible beats rich.\nFigure 1. A learned judge on a short leash; a checker on none.\nWhat the model is actually being taught Here is the part worth sitting with, because it is genuinely strange.\nThe model produces a long chain of reasoning and then an answer. The verifier looks only at the answer. Nothing in the training signal says anything whatsoever about the reasoning — not whether a step was valid, not whether the approach was sensible, not whether it resembles how a person would work. The reward arrives at the very end, attaches to the entire trajectory, and is silent about every token that produced it.\nThis is outcome supervision, and it should by rights be far too crude to work. Imagine grading a thousand-word derivation by writing a single tick or cross at the bottom and handing it back. And yet what emerges from it, without anyone specifying it, is a repertoire of things that look remarkably like thinking: allocating more tokens to harder problems, checking an intermediate result before continuing, noticing an inconsistency and going back, trying a second approach when the first stalls. DeepSeek\u0026rsquo;s report on R1-Zero — a model trained with RLVR directly on a base model, with no supervised demonstrations of reasoning at all — includes a now-famous passage where the model interrupts itself mid-derivation to reconsider. Nobody wrote that behaviour down for it to copy. It appeared because chains that pause to check are, on average, chains that end at the right number.\nThe mechanism behind that emergence is worth being precise about. The reward does not reach back and identify which step was the good one. What it does is raise the probability of every token in a trajectory that succeeded and lower it in every trajectory that failed. Any habit that shows up more often in successful chains than unsuccessful ones will accumulate weight, slowly, across thousands of problems, purely by correlation. Backtracking is not taught. It is filtered for.\nFigure 2. Outcome supervision: a tick at the bottom of the page.\nA pinch of math: judging an answer by the company it keeps The classical machinery for this — PPO , inherited from robotics — carries a second network, a critic, whose job is to predict how much reward a partial trajectory is likely to earn, so that the actual reward can be compared against expectation. A reward of one is only good news if you were expecting less than one. But the critic is roughly the size of the model itself, doubling the memory of training, and predicting the value of a half-finished chain of thought is a hard problem in its own right.\nGroup Relative Policy Optimization, the method behind most current reasoning models, removes it with a trick whose simplicity is the whole of it: instead of predicting what a good score would have been, just sample several answers to the same question and use their average as the yardstick.\nTake a question $q$, sample a group of $G$ answers from the current model — eight, or sixteen, or sixty-four — and let the verifier score each one, giving rewards $r_1 \\ldots r_G$. Then each answer\u0026rsquo;s advantage is simply how it did relative to its peers:\n$$\\hat{A}_i = \\frac{r_i - \\text{mean}(r_1 \\ldots r_G)}{\\text{std}(r_1 \\ldots r_G)}$$If eleven of sixteen attempts failed and this one succeeded, its advantage is strongly positive and every token in it gets pushed up. If it was one of the eleven, everything in it gets pushed down. The group supplies its own baseline, and the critic network is simply gone.\nThe update then nudges the model to make high-advantage trajectories more likely, with one safeguard. For each token, form the ratio between the probability the current model assigns it and the probability the model assigned it when the sample was drawn:\n$$\\rho_{i,t} = \\frac{\\pi_\\theta(o_{i,t})}{\\pi_{\\theta_{\\text{old}}}(o_{i,t})}$$and clip that ratio to a narrow band around one before multiplying it by the advantage. You do not need to trace the algebra; the purpose of the clip is a single conservative instinct. These samples were drawn from a slightly older version of the model, so they are only evidence about that version. Once the current model has moved far enough that a token\u0026rsquo;s probability has changed by more than a small percentage, the clip stops the update from pushing further on the strength of stale evidence.\nOne consequence of the group baseline is worth noticing, because it shapes how these systems are run in practice. If all $G$ answers receive the same reward — every one correct, or every one wrong — then every advantage is zero and the question contributes precisely nothing to the update. Problems the model always solves are wasted compute, and so are problems it never solves. The useful training signal lives entirely in the band of difficulty where the model succeeds sometimes, which is why serious RLVR pipelines spend real effort filtering and re-sorting their problem sets as the model improves. The curriculum is not a nicety here; it is where the gradient comes from.\nFigure 3. Signal lives only where the model succeeds sometimes.\nWhy the search works at all, and what it may not do Step back and notice the assumption underneath everything above. The model can only be rewarded for a correct answer it actually produced. If it never once stumbles onto the right answer for a class of problem, the reward is always zero, the advantages are always zero, and no amount of training will teach it anything. RLVR cannot conjure a capability from nothing. It can only find, amplify, and make reliable the things the base model was already capable of doing occasionally, by luck, in one sample out of a hundred.\nThat framing has led to a real and unsettled argument about what these methods accomplish. There is evidence that if you let a base model take a hundred attempts at a problem and count it solved if any attempt succeeds, the base model and its RLVR-trained descendant are much closer than their single-attempt scores suggest — and, at large enough numbers of attempts, the base model sometimes covers problems its trained descendant no longer reaches. On that reading, training has concentrated probability mass onto the reasoning paths that tend to work, converting a broad and unreliable capability into a narrow and dependable one, while quietly pruning some of the breadth. That is a substantial and valuable thing to have done, and it is not obviously the same thing as having learned to reason better.\nThe counter-argument is that the distinction matters less than it sounds, since a capability you can only access one time in a hundred is not really a capability, and that the gains at longer training runs and harder problem sets do not look purely like sharpening. The honest position today is that both effects are present and their relative size is unresolved. It is worth holding the question open, because a great deal of confident writing about reasoning models assumes an answer to it.\nGaming a verifier that cannot be flattered The reward is incorruptible. The channel around it is not, and this is where practitioners spend most of their time.\nThe verifier only checks what it was written to check. Give a coding problem a weak test suite and the model will discover the function that passes it without solving the problem — special-casing the tested inputs, or finding the degenerate return value that happens to satisfy three assertions out of three. Extract the final answer with a fragile pattern match and the model will learn to produce output that trips the pattern favourably rather than output that is correct. Leave the answer anywhere in the context and it will be found and reproduced without any intervening work; contamination between a training set and a benchmark does the same thing at a larger scale, and is the most common way an impressive number turns out to mean nothing.\nThen there is everything the verifier is indifferent to. R1-Zero, optimized purely for correctness and format, produced chains of thought that switched between languages mid-sentence and were frequently unreadable — not a malfunction, simply the absence of any reward for readability, so none was provided. The eventual R1 added a cold start of supervised examples and a language-consistency term to get a model whose reasoning a person could follow. There is also a persistent pull toward length, since more tokens usually means more chances to recover from an error, and nothing in a correctness reward objects to a model that takes eight thousand tokens to answer something simple.\nThe pattern in all of these is the same. The reward is a perfectly faithful measurement of exactly what it measures, and everything you failed to put in it is fair game. Which is the same lesson as reward hacking in preference learning, arriving from the opposite direction: there, the measurement was unfaithful; here, it is faithful but narrow.\nWhere it stops The requirement is a checker, and most of what we ask models to do does not have one. There is no program that decides whether an essay is well argued, whether an explanation was pitched correctly for its reader, whether a piece of advice was wise, whether a summary kept the right things. For that whole enormous territory, human preferences remain the only instrument we have — whether through a learned reward model, or through direct preference optimization , which dispenses with the reward model but not with the underlying reliance on a proxy for taste. RLVR did not replace preference learning; it carved out the region where something better is available, and the two now run side by side in essentially every frontier training pipeline — verifiable rewards for the domains that admit them, preference learning for taste, safety, and tone.\nThere is steady pressure on that boundary. Process rewards score the individual steps of a derivation rather than only its conclusion, which gives a much denser signal but requires either expensive human labelling of steps or an automatic labeller that is itself a model — reintroducing exactly the proxy we were trying to escape. Rubric-based grading and LLM judges extend the reach into softer domains at the same cost. And in the other direction, work on formalization aims to widen the verifiable region itself, translating informal problems into forms a prover can check. The interesting question for the next few years is how much of human judgement can be turned into something a machine can check without the checker becoming just another opinion.\nFinally, a skill acquired this expensively does not have to be re-acquired. Once a large model has been trained with RLVR, its reasoning traces become supervised data, and a much smaller model trained on them inherits a large fraction of the ability without ever running the search — the reasoning-specific case of distillation , and the reason small open models can now do arithmetic that would have been out of reach for a model of their size two years ago.\nWhat to carry forward RLVR replaces a learned judge with a program that checks the answer: a test suite, a proof checker, a comparison against a known result. Because that reward is a fact rather than an estimate, it cannot be gamed by the tone or shape of a response, which means it can be optimized against far harder than a preference model — and it is that permission to push hard, more than any algorithmic novelty, that produced reasoning models. Supervision falls only on the outcome, never on the reasoning, so behaviours like backtracking and self-checking are not taught but filtered for, accumulating because they appear more often in trajectories that end correctly. The dominant method, GRPO, dispenses with the critic network of classical policy gradients by sampling a group of answers to the same question and scoring each against the group\u0026rsquo;s own average, which also means all the learning signal comes from problems the model solves sometimes — making curriculum difficulty a first-class concern. Two honest caveats travel with all of this: RLVR can only amplify capabilities the base model already exhibits occasionally, and there is real evidence that a meaningful share of its gains is sharpening a distribution rather than extending it; and the reward, while incorruptible, is narrow, so weak tests, leaky answer extraction, benchmark contamination, and unreadable or bloated reasoning are the standing failure modes. It applies only where a checker exists, which is why it now sits alongside preference learning rather than replacing it, and why the live frontier is the question of how much more of human judgement can be made checkable.\nThe figures in this article follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/articles/rlvr/","summary":"\u003cp\u003eAlmost every model in wide use has learned by being shown what a good answer looks like. In pretraining the good answer is simply the next token in the corpus. In \u003ca href=\"/articles/supervised-fine-tuning/\"\u003esupervised fine-tuning\u003c/a\u003e\n it is a demonstration written by a person. In preference learning it is whichever of two responses a human preferred — or, more precisely, whichever a \u003cem\u003ereward model\u003c/em\u003e, itself trained to imitate those humans, predicts a human would have preferred. That last substitution is a compromise the field has quietly lived with for years: the thing doing the grading is a neural network with opinions and blind spots of its own. \u003cem\u003eReinforcement learning from verifiable rewards\u003c/em\u003e, usually shortened to RLVR, is what becomes possible when you can throw the grader away and simply check the answer. This piece is about what changes when the reward stops being a judgement and starts being a fact.\u003c/p\u003e","title":"RLVR — Letting the Answer Key Do the Teaching"},{"content":"The machinery of attention is simple enough to state: every word in a sentence gets to look at every other word, weigh how much each one matters, and gather in what it needs. It is a flexible mechanism, and it hides a blind spot. Left to itself, attention has no idea what order the words came in. A sentence arrives as a handful of scrabble tiles tipped onto a table: all the pieces are there, and nothing says which came first. \u0026ldquo;Dog bites man\u0026rdquo; and \u0026ldquo;man bites dog\u0026rdquo; are, to raw attention, the same pile. Since word order carries a great deal of meaning, something has to hand the model the seating chart. Rotary Position Embedding — RoPE — is how most current models do it, and the move is physical: it tells each word where it sits by rotating it. This piece is about why that works, what it buys at longer lengths, and the one thing it does not fix.\nWhy order has to be supplied at all It is worth pausing on the blind spot, because everything follows from it. Inside attention, each word is turned into a bundle of numbers — a vector, an arrow pointing somewhere in a high-dimensional space — and words compare themselves to one another by measuring how aligned their arrows are. That comparison does not depend on where in the sentence each word happened to appear. Swap the first and last word and the arrows are unchanged, so the comparisons come out identical. The mechanism is permutation-blind by construction. This is not a bug so much as a consequence of how the math is built, but it means that if the model is to know that \u0026ldquo;not\u0026rdquo; sits before \u0026ldquo;happy\u0026rdquo; rather than after it, the information has to be injected deliberately. Every positional scheme ever invented is a different answer to one question: how does each word learn where it sits?\nThe tempting first answer, and why it strains The obvious idea is to stamp each word with its seat number — position one, position two, position three — by adding a position-specific pattern to its vector before attention sees it. This is the original approach, and it works. It also has a nagging weakness. The model learns what each seat number means only from the lengths it saw during training. Show it a text longer than anything it trained on and it meets seat numbers it has never encountered, and its sense of position frays. There is something wasteful about it too: language rarely cares about a word\u0026rsquo;s absolute seat. What matters is almost always relative — \u0026ldquo;the adjective two words before this noun,\u0026rdquo; \u0026ldquo;the subject back at the start of the clause.\u0026rdquo; That relationship is the same whether the phrase lands near the beginning of a paragraph or near the end. A scheme built around absolute seat numbers encodes the wrong thing and then hopes the model can recover the thing it wanted. RoPE encodes the right thing directly.\nThe core idea: rotate each word by its position Here is the move at the heart of RoPE. Take each word\u0026rsquo;s vector and, before attention compares it with the others, rotate it by an angle that depends on its position in the sentence. The word in position one is turned a little; the word in position two, twice as much; position three, three times as much, and so on. Picture each word\u0026rsquo;s arrow lying on a clock face: RoPE spins the first word\u0026rsquo;s hand forward by one tick, the second word\u0026rsquo;s by two ticks, the tenth word\u0026rsquo;s by ten.\nAt first this looks as though it would reintroduce the absolute-position problem in a new costume, since each word is being turned by its own absolute seat number. What matters is what happens when two rotated words then compare themselves. Attention measures alignment between two arrows. Rotate two arrows and then measure how aligned they are, and the answer depends only on the difference between the two rotations, not on either rotation on its own. Turn both arrows by the same extra amount and their relative angle is untouched. So a word at position 5 and a word at position 8, each spun by its own absolute position, compare in a way that depends only on the gap — three positions apart — and come out identical for a pair at positions 105 and 108. The rotation is by absolute position, and what survives the comparison is purely relative distance. That is the whole trick: the simplicity of stamping each word independently, with the robustness of caring only about gaps, at the price of a pair of sines and cosines.\nFigure 1. Rotating both arrows leaves only the gap.\nA pinch of math: rotation is just a small turn The idea is easy to make precise. Take a word\u0026rsquo;s vector and chop it into pairs of numbers — the first two, the next two, and so on. Each pair is treated as a two-dimensional arrow, a point on its own dial. To rotate a two-dimensional arrow $(x_1, x_2)$ by an angle $\\theta$, apply the standard rotation:\n$$\\begin{pmatrix} x_1' \\\\ x_2' \\end{pmatrix} = \\begin{pmatrix} \\cos\\theta \u0026 -\\sin\\theta \\\\ \\sin\\theta \u0026 \\cos\\theta \\end{pmatrix} \\begin{pmatrix} x_1 \\\\ x_2 \\end{pmatrix}$$That is all a rotation is: mix the two numbers together using sines and cosines so the arrow swings around the dial without changing length. RoPE sets the angle to be the word\u0026rsquo;s position $m$ times a fixed base rate, so the pair is turned by $m\\theta$. Now the payoff. When a word at position $m$ compares with a word at position $n$, the comparison for each pair works out to depend on the angle $(m - n)\\theta$ — the difference of their positions. The absolute positions $m$ and $n$ enter only through their difference. The interaction between two words becomes a function of $m - n$ alone, which is the relative-distance behaviour wanted, falling out of the arithmetic of rotation. This is the property the additive schemes could only approximate: there, position was added to content, the score expanded into four terms, and only one of them was a clean function of the gap. Here nothing else survives the comparison.\nFast and slow dials: caring about near and far at once One more ingredient makes RoPE work in practice, and it explains a detail passed over above. Not every pair of numbers is rotated at the same rate. RoPE assigns each pair its own turning speed — some dials spin quickly as position advances, others turn slowly. The fast dials complete a full revolution over a few positions, so they are sensitive to fine, local distances: is this word right next to that one, or two away? The slow dials take hundreds or thousands of positions to come around, so they track coarse, long-range structure: are these two words in the same paragraph, or pages apart?\nThe hands of a clock divide labour the same way. A second hand resolves seconds and is useless for months; an hour hand gives the rough time of day and resolves nothing finer. By spinning many dials at many speeds at once, RoPE hands the model a multi-scale sense of distance — sharp up close, gentle far away — from the same act of rotation. This spread of frequencies is why RoPE captures both phrase-level and passage-level structure without any of it being learned or stored.\nFigure 2. Many dials, many speeds.\nNothing to store, nothing to run out of Step back and notice what RoPE does not need. There is no table of position patterns to learn, so there is nothing that runs out when a text grows longer than the training texts. The rotation for position one-thousand-and-one is computed the same way as the rotation for position five, by turning through a larger angle. Because the scheme is a mathematical function of position rather than a lookup of memorized values, it extends past its training lengths more gracefully than the old stamp-a-seat-number approach. It costs almost nothing to store and little to run: rotating a handful of number-pairs is cheap. It also acts on the query and key vectors before any score matrix exists, which leaves the fused attention kernels of FlashAttention undisturbed. A scheme that adds a bias to the score grid has to index into that grid instead, and the article on positional encoding records the consequence: a positional scheme that obstructs the fastest available kernel tends not to get adopted.\nFigure 3. Learned table versus computed rotation.\nStretching a model to longer texts RoPE opens a practical door, and it is the reason headlines keep appearing about models that support a context of hundreds of thousands of words. A model trained on short texts can be pushed to read much longer ones by adjusting the dials rather than retraining from scratch. The intuition is spatial: compress the new positions so far-apart words appear closer together, back inside the range of angles the model already understands, or rescale the turning speeds selectively, leaving the fast local dials sharp while stretching only the slow long-range ones. With a small amount of extra training, and occasionally none, a model built for modest lengths reads very long documents. It amounts to re-spacing the seats so an oversized room presents the angles of the familiar one, and it is much of the machinery behind the context windows that keep growing.\nThe rescaling is a trade, and it is worth naming as one. Compress every dial equally and neighbouring tokens become harder to tell apart, because the fine dials now turn through smaller arcs between adjacent words — which is exactly why the schemes that work rescale selectively, stretching the slow dials and leaving the fast ones alone. Reach is bought from local resolution, and the craft is in choosing where to spend it. Nor does RoPE extrapolate unaided. Run a model past its trained lengths without rescaling and quality falls away, which is the finding the article on positional encoding reports for every scheme in use, RoPE included. What the rescaling buys is a window, not the ability to use it. The dilution of attention across a longer context and the linear growth of the KV cache are untouched by anything done to the dials, and the article on long context treats both. Position is one constraint of several, and the only one rotation addresses.\nWhat to carry forward Attention is blind to word order, so order must be supplied, and language nearly always cares about relative position rather than absolute seat numbers. RoPE meets both facts with one move: it rotates each word\u0026rsquo;s vector by an angle proportional to its position, and because the comparison between two rotated words depends only on the difference of their rotations, what emerges is relative distance. Splitting the vector into many pairs and spinning each at its own speed supplies a multi-scale sense of near and far at once, and because the whole thing is a computed function of position rather than a stored table it neither runs out at new lengths nor costs much to use, and it acts early enough to leave the fused kernels alone. It displaced the earlier schemes on those properties rather than on accuracy. What it does not supply is extrapolation. Past the trained lengths the dials must be rescaled, and rescaling buys reach out of local resolution unless it is done selectively. Position was also only ever one of the constraints: attention still dilutes across a long window, and the cache still grows with every token. Rotation does not touch either.\nThe figures in this article follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/articles/rope/","summary":"\u003cp\u003eThe machinery of \u003ca href=\"/articles/attention/\"\u003eattention\u003c/a\u003e\n is simple enough to state: every word in a sentence gets to look at every other word, weigh how much each one matters, and gather in what it needs. It is a flexible mechanism, and it hides a blind spot. Left to itself, attention has no idea what \u003cem\u003eorder\u003c/em\u003e the words came in. A sentence arrives as a handful of scrabble tiles tipped onto a table: all the pieces are there, and nothing says which came first. \u0026ldquo;Dog bites man\u0026rdquo; and \u0026ldquo;man bites dog\u0026rdquo; are, to raw attention, the same pile. Since word order carries a great deal of meaning, something has to hand the model the seating chart. \u003cem\u003eRotary Position Embedding\u003c/em\u003e — RoPE — is how most current models do it, and the move is physical: it tells each word where it sits by \u003cem\u003erotating\u003c/em\u003e it. This piece is about why that works, what it buys at longer lengths, and the one thing it does not fix.\u003c/p\u003e","title":"RoPE — Telling a Model Where a Word Sits by Turning It"},{"content":"A model that turns down a request appears to be exercising judgement, and the wording encourages that reading, since it tends to say it cannot help with that, as though something in it were unable rather than unwilling. Nothing in the system is unable. The material that would answer the question sits where it sat before anyone trained the model to decline, and what was added on top is a behaviour — a learned tendency to produce one kind of continuation rather than another, installed by the same optimizer as everything else and heir to the same failures. Reading refusal as a capability removed rather than a behaviour added is the most common misunderstanding here, and nearly every surprising result follows from getting it the right way round. This piece is about how refusal is actually trained, why its two failure modes trade against each other, and why the behaviour is thinner than it looks.\nPretraining produces no reluctance The constraint underneath all of this is worth stating precisely, because it is usually stated loosely. A base model is trained to predict the continuation of human text, so whatever is present in that text is something it was rewarded for reproducing accurately. Instructions for harm are there, along with fraud, abuse, and every technical detail anybody wrote down. The objective contains no term separating material someone would want reproduced from material nobody would: it is one number measuring how surprised the model was, and surprise has no ethics.\nThe base model does not decline because it has no notion of a request. It has a notion of a continuation. Given something shaped like a question it produces something shaped like an answer, not because it decided to help but because in the text it learned from, questions are followed by answers; given a manual it continues the manual. The option of not continuing never arises. Refusal is not a property that survives pretraining and then degrades under pressure. It was never there.\nRefusal is trained by the machinery that trains everything else What gets added afterwards uses no new mechanism, and that is the most useful single fact for an engineer to hold.\nIt arrives in two stages, neither of them special. During supervised fine-tuning the model is shown demonstrations of declining — prompts the operator does not want answered, paired with responses that decline them — under the same loss as the demonstrations teaching it to summarize a document. Then comes preference training, where two responses to one prompt are ranked and the decline is preferred. Those rankings are either compressed into a reward model and optimized against with PPO , or handed to DPO directly. Either way the machinery is indifferent to what the preference is about. A pair whose chosen response declines to help with a weapon is processed identically to one whose chosen response merely has tidier formatting.\nBecause the mechanism is ordinary, the pathologies are ordinary too. Every failure mode of preference learning applies here unmodified. The data is annotated by people whose thresholds disagree, so the target is noisy before any optimization begins. Methods that optimize a margin between chosen and rejected can lower the probability of both, and the displaced mass has been seen landing on the wrong side: a preferred refusal made less likely, a compliance more so. Reward models can be satisfied by the surface features of a decline rather than the decision behind it, which is one reason production refusals all sound alike; a register is easier to reproduce than a judgement. Refusal is not a safety module bolted to the side of a language model. It is one more preference, fitted like the rest.\nFigure 1. Where the decline gets installed.\nTwo errors, one dial, and no honest way to measure either A classifier that never errs does not exist, and refusal is a classifier whatever else it may be. So training does not move toward correctness. It moves along a curve, and both ends are bad.\nAt one end the model declines what it should not — over-refusal. A clinician asking about lethal doses is asking a question the job requires. A novelist asking for a violent scene is asking for fiction, which has always contained violence. A defender asking how an attack works is asking the question that precedes fixing it. All three get refused regularly, and the cause is not prudishness but the shape of the signal: they share surface features with the requests the model was taught to decline, while the context that would separate them — who is asking, and why — is absent or unverifiable.\nAt the other end it complies with what it should not — under-refusal — which needs no illustration.\nThe two rates are not independently adjustable. There is roughly one dial, and turning it down to rescue the clinician turns it up for everybody else. Where to leave it depends on the users, the deployment, and what the operator will answer for — a product decision, taken by engineers, habitually described as a technical one.\nWorse, the position cannot be located empirically. Refusal benchmarks have a shorter useful life than most: publishing prompts a model ought to decline puts them where the next training run will scrape them, after which the score reports recall of specific items rather than the behaviour they sampled. The quantity scored is a judgement rather than a fact, so scoring means another model\u0026rsquo;s opinion about whether a response was a refusal and whether it should have been. And the base rate is decisive: harmful requests are a small fraction of real traffic, so a benchmark balanced between harmful and benign resembles nothing in deployment, and a false-positive rate that looks tolerable there can dominate ordinary users\u0026rsquo; experience. These are the standing problems of evaluation in an unfavourable configuration.\nA pinch of math: the overlap sets the floor Reduce the arrangement to its simplest honest form. Suppose we have a scalar $s$ computed from a request, standing in for however the model represents the question of whether to comply, and a threshold $\\tau$ above which it declines. Harmful requests are drawn from one distribution over $s$ and benign requests from another, and the two overlap. The error rates are then integrals over opposite tails:\n$$\\text{under-refusal} = \\int_{-\\infty}^{\\tau} p_H(s)\\, ds, \\qquad \\text{over-refusal} = \\int_{\\tau}^{\\infty} p_B(s)\\, ds$$Moving $\\tau$ moves both, in opposite directions, and no value of it makes both small while the distributions overlap. There is one number to set and two quantities that depend on it.\nWhat the picture makes visible is what training is actually for. Turning $\\tau$ is free and buys nothing. The quantity deciding whether any threshold can work is the overlap, and the minimum total error is fixed by it alone — the Bayes error of the problem, which no decision rule beats. So the work that matters is separating the distributions: giving the model whatever it needs to place the clinician\u0026rsquo;s question and an identical-sounding one from someone else in different regions of $s$. Where two requests are indistinguishable from their text alone, the overlap is irreducible, and so is the error.\nFigure 2. One threshold, two tails, and the overlap.\nThe behaviour is thin, and jailbreaks work on the thinness One line of interpretability work bears on how deep the installed behaviour goes. In open-weight models, refusal appears to be mediated by a comparatively simple internal signal — close to a single direction in the residual stream , the running vector each layer reads from and writes to. That direction can be identified and suppressed, and when it is the model stops declining while ordinary task performance is largely unchanged. The finding matters for what it implies rather than for what it enables. Refusal was a thin behavioural layer over a model otherwise left alone. The knowledge is still in there. Training added a disposition rather than removing an ability. This is one line of evidence, not a settled account — a single clean direction is probably too tidy, given how features in these models share directions in the way the study of superposition describes — but it points consistently.\nThat thinness is the background against which refusal fails under pressure, and the account of the failures is structural. Three causes, each a class rather than a technique.\nDistribution shift. Refusal was fitted on inputs resembling the safety training data, and like any learned behaviour generalizes worse the further an input sits from those. Anything that makes a request unlike those examples moves it where the behaviour was never trained, and what the model does there is whatever its general tendencies produce. This is not a special weakness of safety data. It is the ordinary generalization gap, on a behaviour where gaps are expensive.\nCompeting objectives. The same post-training that installs refusal also installs helpfulness, instruction-following, and a pull toward completing whatever pattern has been started. Those are separate pressures on one set of weights, and can be put into tension, so that declining requires the model to violate something else it was trained to do. The model is not being talked out of its training. It is being placed where two parts of its training disagree, and something gives.\nThe shared channel. Instructions and data arrive as one undifferentiated stream of tokens. Nothing in the input marks a passage as material to be processed rather than direction to be followed, and nothing in the architecture could enforce such a mark. Content the model was asked to summarize can therefore be read as content addressing it. This is the fact underneath prompt injection, and the reason injection is unfixable rather than merely unfixed: there is no parser to patch. The exposure grows with anything that widens the input: retrieved context, tool output, other agents.\nNone of these closes. Each is answered by training on the cases that got through, which narrows that region and shifts the frontier elsewhere, against an adversary who adapts faster than a training run completes. Refusal is a moving equilibrium maintained by continuous effort, not a property established once, and underneath it the model\u0026rsquo;s capabilities are exactly what they were.\nFigure 3. A thin layer over untouched capability.\nWhat to carry forward Nothing in pretraining knows how to decline, because predicting the continuation of human text carries no notion of a request that should go unanswered; refusal is installed afterwards, by demonstrations during supervised fine-tuning and then by preference pairs marking a decline over a compliance, run through a reward model and reinforcement learning or through direct preference optimization. No new mechanism appears in that, which is why every pathology of preference learning is a pathology of refusal: noisy human thresholds, margin objectives that displace probability onto responses nobody chose, reward models satisfied by the register of a decline rather than the decision behind it. Refusal has two errors that trade — over-refusal, which turns away the clinician, the novelist, and the defender for resembling requests in the training data, and under-refusal, which needs no illustration — governed by roughly one dial whose setting is a product decision in technical language. Formally the two rates are tails of overlapping distributions either side of one threshold and cannot be set independently, and the floor on total error is fixed by the overlap rather than the threshold, so the only work that helps is whatever separates them. Measurement is the weakest link: refusal benchmarks become training data on publication, the thing scored is a judgement usually another model\u0026rsquo;s, and real traffic contains too few harmful requests for a balanced benchmark to describe it. And the layer is thin: in open-weight models the behaviour appears to ride on a compact internal signal that can be suppressed with little loss of capability, which fits the three structural reasons jailbreaks work — distribution shift, objectives in tension, and a token stream that cannot distinguish instruction from data. What is underneath has not been altered, and the layer above it holds only as long as it is maintained.\nThe figures in this article follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/articles/safety-and-refusal/","summary":"\u003cp\u003eA model that turns down a request appears to be exercising judgement, and the wording encourages that reading, since it tends to say it \u003cem\u003ecannot\u003c/em\u003e help with that, as though something in it were unable rather than unwilling. Nothing in the system is unable. The material that would answer the question sits where it sat before anyone trained the model to decline, and what was added on top is a behaviour — a learned tendency to produce one kind of continuation rather than another, installed by the same optimizer as everything else and heir to the same failures. Reading refusal as a capability removed rather than a behaviour added is the most common misunderstanding here, and nearly every surprising result follows from getting it the right way round. This piece is about how refusal is actually trained, why its two failure modes trade against each other, and why the behaviour is thinner than it looks.\u003c/p\u003e","title":"Safety and Refusal — Teaching a Model to Decline"},{"content":"There is something peculiar about the fact that this works at all. A neural network is a chaotic object — billions of parameters set by a stochastic process, on data nobody has fully inspected, through an optimization that visits an incomprehensible landscape. Nothing about that description suggests predictability. And yet the loss such a model achieves is a smooth, regular function of three numbers: how large it is, how much it read, and how much arithmetic was spent. Plot loss against any of them on logarithmic axes and you get a straight line, holding over many orders of magnitude. This is what makes it possible to spend a hundred million dollars on a training run with any confidence about the outcome, and it is the closest thing the field has to a physical law. This piece is about the relationship, the correction that reshaped the field\u0026rsquo;s spending, and the ways the law is more limited than its elegance suggests.\nThe shape of the relationship Hold everything else fixed and vary one quantity — parameters $N$, training tokens $D$, or total compute $C$ — and the loss follows a power law with an offset:\n$$L(N) = \\frac{A}{N^{\\alpha}} + L_\\infty$$Two parts, and both matter. The power-law term is the reducible loss, and it falls steadily but with diminishing returns: each halving of the loss above the floor costs a multiplicative increase in scale, not an additive one. The offset $L_\\infty$ is the irreducible loss — the entropy of language itself, as the article on cross-entropy described, which no model can cross because text is genuinely not deterministic.\nThe remarkable properties are the empirical ones. The exponents are small, which means progress is slow but extraordinarily reliable — though the precise values depend entirely on how the curve was fitted, and quoting them across papers is a trap. Fits that absorb the irreducible floor into the power law report exponents around $0.05$ to $0.1$; fits that subtract an explicit floor first, and so describe only the reducible part, report around $0.3$. The two are not comparable, and most published disagreement about \u0026ldquo;the\u0026rdquo; exponent is this distinction rather than a disagreement about the data. The relationship holds across many orders of magnitude with very little scatter. And it is largely insensitive to architectural detail — depth against width, the precise attention variant, most of the choices practitioners argue about move the curve by a small constant factor rather than changing its slope. Scale dominates, and it dominates by so much that the correct first response to \u0026ldquo;will this architectural change help?\u0026rdquo; is usually \u0026ldquo;less than making the model bigger.\u0026rdquo;\nFigure 1. A straight line and the floor beneath it.\nThe correction that moved a billion dollars The first influential formulation, from 2020, examined how a fixed compute budget should be divided between model size and training data, and concluded that model size deserved most of it. Roughly, if compute increases tenfold, make the model about five times larger and train it only about twice as long. The industry followed, and the era of enormous, briefly-trained models followed with it: models of a hundred and seventy-five billion parameters trained on a few hundred billion tokens, barely more than one token per parameter.\nTwo years later a careful re-examination found this was wrong, and the reasons are a lesson in themselves. The correcting paper pointed to a learning-rate schedule that did not decay properly for shorter runs, making those runs look worse than they were and so making training longer look less valuable than it was. Later work has revised that diagnosis: the largest single factor appears to be that the original counted non-embedding rather than total parameters, and fitted at small scale where that distinction matters most, with warm-up duration and scale-dependent optimizer settings contributing the rest. The specific culprit is still argued over. What is not argued over is the shape of the failure — a set of methodological choices in ancillary settings, none of them the object of study, misdirected the field\u0026rsquo;s largest expenditure for two years.\nThe corrected result is that parameters and tokens should scale in equal proportion. Double the compute and you should make the model about $1.4$ times larger and train it on about $1.4$ times more data — not five times larger and twice as long. The compute-optimal ratio works out to roughly twenty tokens per parameter, and the demonstration was direct: a seventy-billion-parameter model trained on 1.4 trillion tokens outperformed a model four times its size across the board, using the same compute.\nThe reason the arithmetic comes out this way is straightforward once seen. Training compute is approximately\n$$C \\approx 6ND$$— six floating-point operations per parameter per token, two for the multiply-add in the forward pass and roughly twice that again for the backward pass. So $N$ and $D$ enter the budget symmetrically. If the loss also responds to them with similar exponents, which it approximately does, then symmetry in the cost and symmetry in the benefit means they should be scaled together. The earlier asymmetric recommendation had implied the exponents were quite different, and the flawed schedule was what created that impression.\nFigure 2. Five times larger, or one point four each.\nThe twist: training-optimal is not serving-optimal The correction was quickly followed by a realization that complicates it, and that now governs how most deployed models are built.\nCompute-optimal answers the question what is the best model I can train for this budget. Almost nobody actually wants the answer to that question. What they want is the best model they can afford to run, and inference cost is paid per request, forever, by everyone. A smaller model is cheaper on every single query it will ever serve.\nSo it is rational to deliberately overshoot the token count — to train a smaller model on far more data than the compute-optimal ratio recommends, accepting a worse loss for the training budget in exchange for a permanently cheaper model. The loss curve is quite flat in this region, so the sacrifice is modest and the saving is not. This is why models are now routinely trained at hundreds or even thousands of tokens per parameter, an order of magnitude past twenty. The Chinchilla ratio is the right answer to a question that most people are not asking.\nThe correct framing is total cost of ownership: training compute plus expected lifetime inference compute. Under that objective the optimal model is smaller and better-read than the compute-optimal one, and how much smaller depends on how many requests you expect to serve — which is a business forecast rather than a scientific quantity.\nFigure 3. Total cost of ownership, not training cost.\nWhat the law does not tell you Four honest limitations, because the relationship is often extended past where it has been established.\nIt predicts loss, not capability. The smooth curve describes cross-entropy on held-out text. What people care about is whether the model can do things, and the mapping from one to the other is not smooth. Performance on specific tasks has repeatedly appeared to jump at particular scales rather than improving gradually — the phenomenon called emergence . There is a good deflationary argument that much of this is an artifact of measurement: a metric like exact-match accuracy is a threshold on an underlying continuous quantity, and a smoothly improving probability of getting every step right will look like a sudden jump when you only score complete successes. Where the sharpness is real and where it is the scoreboard remains genuinely contested, and either way the practical point stands — the loss curve will not tell you at what scale a capability appears.\nThe exponents are properties of the data. A scaling law is fitted on a particular corpus and describes that corpus. Change the data mixture , the filtering strictness, or the repetition rate and the constants move. Improving the data shifts the entire curve down, which is a more valuable thing to do than moving along it, and is invisible to a law fitted before the change.\nRepetition breaks it. The relationship assumes fresh tokens. As discussed under data mixtures, repeated data is nearly as good as new up to about four passes and rapidly less so thereafter, so the token axis stops meaning what it meant once you are recycling.\nIt says nothing about post-training. Everything here concerns pretraining loss. Reasoning models added a second axis entirely — compute spent at inference time, which trades against model size in a way no pretraining law anticipated — and the scaling behaviour of reinforcement learning is a separate and much less settled empirical question.\nA pinch of math: reading the exponent One calculation is worth being able to do, because it converts the abstract exponent into an intuition about cost.\nTake the reducible part of the loss, $L \\propto C^{-\\beta}$, and use the small end of the range above, $\\beta \\approx 0.05$. Ask what it costs to halve the reducible loss. Setting $2 = (C_{\\text{new}}/C_{\\text{old}})^{\\beta}$ and solving:\n$$\\frac{C_{\\text{new}}}{C_{\\text{old}}} = 2^{1/\\beta} = 2^{20} \\approx 10^{6}$$A million-fold increase in compute to halve the reducible loss. This is the number that puts the whole enterprise in perspective. Progress in this direction is real, and it is reliable enough to plan a datacentre around, and it is brutally expensive — which is precisely why so much effort goes into everything that shifts the curve rather than travelling along it: better data, better architectures, distillation, and spending compute at inference instead of in training. Scaling works. It just does not work cheaply, and every alternative route to the same loss is worth an enormous amount.\nWhat to carry forward The loss a language model reaches is a strikingly regular function of its size, its data, and its compute — a power law plus an irreducible floor set by the entropy of language, straight on logarithmic axes across many orders of magnitude, and largely indifferent to the architectural details practitioners argue about. That regularity is what makes an enormous training run plannable. The influential first formulation concluded that compute should go mostly into model size, and was wrong for methodological reasons still being argued over — a learning-rate schedule that penalized short runs, and, on later analysis more importantly, counting non-embedding rather than total parameters while fitting at small scale; the corrected result is that parameters and tokens should scale together, at roughly twenty tokens per parameter, which follows naturally from training compute being about six operations per parameter per token and therefore symmetric in the two. But compute-optimal answers a question few people are asking, since inference is paid forever and a smaller model is cheaper on every request — so deployed models are deliberately trained well past the optimal ratio, accepting a slightly worse loss for a permanently cheaper model. The law\u0026rsquo;s limits are worth respecting: it predicts loss rather than capability, and the mapping between them is not smooth; its constants are properties of a particular corpus, so improving the data moves the whole curve rather than moving along it; it assumes fresh tokens and degrades under repetition; and it says nothing about post-training or about inference-time compute, which is a separate axis entirely. And the exponent is small enough that halving the reducible loss costs something like a million-fold increase in compute — which is the honest reason so much ingenuity is spent on anything that shifts the curve instead.\nThe figures in this article follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/articles/scaling-laws/","summary":"\u003cp\u003eThere is something peculiar about the fact that this works at all. A neural network is a chaotic object — billions of parameters set by a stochastic process, on data nobody has fully inspected, through an optimization that visits an incomprehensible landscape. Nothing about that description suggests predictability. And yet the loss such a model achieves is a smooth, regular function of three numbers: how large it is, how much it read, and how much arithmetic was spent. Plot loss against any of them on logarithmic axes and you get a straight line, holding over many orders of magnitude. This is what makes it possible to spend a hundred million dollars on a training run with any confidence about the outcome, and it is the closest thing the field has to a physical law. This piece is about the relationship, the correction that reshaped the field\u0026rsquo;s spending, and the ways the law is more limited than its elegance suggests.\u003c/p\u003e","title":"Scaling Laws — Predicting Quality Before You Train"},{"content":"The softmax turns up nearly everywhere in a language model, almost always as a piece of settled machinery: the thing that turns scores into odds. It deserves a closer look, because it is not a neutral conversion. It imposes a condition — that the results must sum to one — and that condition, harmless when the question is which word comes next, becomes a genuine constraint when the same function is used inside attention, where the honest answer is sometimes that nothing here is worth attending to. A model given no way to say nothing will say something anyway, and where it puts that unwanted output turns out to explain a scattering of otherwise baffling observations: strange spikes in attention on the first token of every sequence, outliers that ruin quantization, and models that collapse when their oldest tokens are discarded. This piece is about what the softmax insists on, and what a model does when it cannot comply.\nWhat the softmax insists on The function takes a list of real numbers and returns a list of positive numbers summing to one:\n$$p_i = \\frac{e^{z_i}}{\\sum_j e^{z_j}}$$Two properties matter, and it is worth separating them, because only one of them is usually wanted.\nThe first is the soft part, which is the useful part. Exponentiating and normalizing produces a smooth, differentiable stand-in for taking the maximum. A hard maximum has no useful derivative — nudge the runner-up and nothing changes at all until it overtakes, at which point everything changes at once — and a network cannot be trained through such a thing. The softmax gives a version where every candidate\u0026rsquo;s weight responds continuously to every score. That is what the name means: a soft argmax.\nThe second property is the normalization, and it is a constraint rather than a convenience. Whatever the inputs, the outputs sum to exactly one. Feed it a list of enormous numbers and it returns a distribution; feed it a list of tiny negative ones and it returns a distribution. There is no input for which the softmax outputs something small. The total is fixed in advance, and only its allocation is up for negotiation.\nFor the output of a language model this is exactly right. Something must come next, so a distribution over candidates is the correct object and the constraint encodes a true fact about the situation.\nInside attention, it is not right at all.\nThe problem: attention cannot abstain Recall how attention uses the softmax. Each position scores every position it is allowed to see, and those scores are passed through a softmax to produce weights. Because the weights sum to one, every position takes a full unit of attention from somewhere, on every head, in every layer, without exception.\nNow consider what a head is actually for. A head that tracks the subject of a verb has nothing to do while processing a token that is not a verb. A head that matches closing brackets has nothing to do when no bracket is open. A head that resolves pronouns has nothing to do most of the time. These are not malfunctions; a specialized component is idle whenever its speciality is absent, and in a model with a thousand heads most of them are idle at any given position.\nAn idle head would like to contribute nothing. It cannot. The softmax obliges it to distribute a full unit of weight across the positions available, and whatever values those positions hold will be blended and written into the residual stream . The head has no way to express not applicable. It must attend to something, and whatever it attends to it will act upon.\nSo the model faces a problem it did not choose and must solve with the tools it has. It needs somewhere to dump attention it does not want to use — a position that can safely absorb arbitrary amounts of weight without contributing anything harmful when its value is mixed in.\nAttention sinks: where the leftover goes What models converge on, reliably and without anyone designing it, is to nominate a few positions as dumping grounds. These are the attention sinks, and the usual choice is the very first token of the sequence.\nThe first token is an inspired candidate for the role. It is visible to every position under a causal mask, which no other token is; it is present in every sequence, so the arrangement generalizes; and being first, it has the least context of any token, so it is the least useful one to have been carrying real information. During training the model learns to give it a value vector of very small magnitude, so that mixing it in changes the stream hardly at all, while giving it a key that scores well against the queries of any head with nothing better to do.\nOnce you know to look, the phenomenon is unmissable. Plot the attention weights of a trained model and a bright vertical stripe runs down the first column — often more than half of all attention mass in the upper layers, concentrated on a token that is frequently just a start marker or the word \u0026ldquo;The\u0026rdquo;. For years this was read as a curiosity or a bug. It is neither. It is the escape valve the architecture failed to provide, improvised out of whatever was available.\nFigure 2. The bright stripe on the first column.\nA pinch of math: the missing option The cleanest way to see the problem is to write down what the model would need and notice it is not there.\nGiven scores $z_1 \\ldots z_n$ over the positions available, attention computes\n$$w_i = \\frac{e^{z_i}}{\\sum_{j=1}^{n} e^{z_j}}$$and $\\sum_i w_i = 1$ identically. Now suppose we add a single extra term to the denominator, corresponding to an option that always scores zero and carries no value:\n$$w_i = \\frac{e^{z_i}}{1 + \\sum_{j=1}^{n} e^{z_j}}$$Since $e^0 = 1$, that added one is a phantom candidate: always present, always with a score of zero, contributing nothing when selected. Now the weights sum to something strictly less than one, and the shortfall is the weight assigned to the phantom. If every real score is well below zero — nothing here is worth attending to — the denominator is dominated by the one, every real weight is small, and the head\u0026rsquo;s output is close to nothing at all. The head can abstain.\nThis modification has been proposed several times, under names like softmax-with-one or quiet attention, and it is a two-character change to the code. Its interest is less as a fix than as a diagnosis: it makes precise what the standard formulation withholds, which is the ability to output a small vector. A model without it must manufacture an equivalent, and the sink token is that manufacture — a real position pressed into service as the phantom, with a near-zero value learned to make it behave like one.\nFigure 1. The phantom option the denominator lacks.\nWhat this explains The oddity has consequences well beyond the attention map, and they are the reason it repays this much attention.\nStreaming breaks without it. If you run a model over an indefinitely long stream and keep only a sliding window of recent tokens, the natural implementation discards the oldest, including the first. Do that and quality collapses immediately — not degrades, collapses. The reason is now obvious: the sink is gone, the attention that had been parked on it is forcibly redistributed across the remaining tokens, and every head that had been idling is suddenly acting on whatever it lands on. The fix is to keep the first handful of tokens permanently, regardless of how far the window has moved on. A few tokens of ballast, retained forever, restore ordinary behaviour. This is the practical result that made the phenomenon widely known, and it is why the eviction schemes described in the article on the KV cache always exempt the beginning of the sequence.\nQuantization suffers. To function as a sink, a token needs a key that scores far above everything else, and the model produces that by developing very large values in a few particular dimensions of the stream. Those outliers are enormous — orders of magnitude above the typical activation — and they wreck any quantization scheme that assigns a shared scale to a whole tensor, because the scale must accommodate the outlier and everything else is then crushed into a handful of levels. A good deal of the difficulty in quantizing activations, and several of the techniques invented to work around it, trace back to dimensions that exist to support an attention sink.\nVision models grow them too. The same phenomenon appears in transformers trained on images, where a few uninformative background patches acquire abnormally high attention and large activations, disrupting the feature maps people wanted to use for segmentation. The fix that worked was to append a few extra learned tokens with no image content attached — registers, whose only job is to be attended to. Giving the model somewhere to put its spare attention made the artifacts disappear, which is about as direct a confirmation of the diagnosis as one could ask for.\nFigure 3. One constraint, three downstream symptoms.\nWhat to carry forward The softmax does two separable things: it provides a differentiable stand-in for the maximum, which is what anyone wants, and it forces its outputs to sum to exactly one, which is a genuine claim about the situation. For a model\u0026rsquo;s final distribution over the vocabulary that claim is true, since something must come next. Inside attention it is false, because a specialized head is idle most of the time and would like to contribute nothing — yet the normalization obliges every head, at every position, to take a full unit of attention from somewhere and act on whatever it takes. Models resolve this by electing an attention sink, almost always the first token, which is visible to all positions, present in every sequence, and can be given a near-zero value so that mixing it in does no damage; the vertical stripe of attention on the first token that shows up in every attention map is this arrangement, not an anomaly. A two-character change to the denominator — adding a phantom always-zero option — would supply the escape valve directly, and its real value is diagnostic: it names exactly what the standard formulation withholds. The consequences of the improvisation are concrete. Discarding early tokens in a sliding window destroys the sink and collapses quality, which is why streaming schemes pin the first few tokens permanently. The extreme activations required to make a sink work are a major source of the outliers that make activation quantization hard. And the same pathology in vision transformers was cured by appending spare register tokens whose only function is to absorb attention that had nowhere else to go.\nThe figures in this article follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/articles/softmax/","summary":"\u003cp\u003eThe softmax turns up nearly everywhere in a language model, almost always as a piece of settled machinery: the thing that turns scores into odds. It deserves a closer look, because it is not a neutral conversion. It imposes a condition — that the results must sum to one — and that condition, harmless when the question is \u003cem\u003ewhich word comes next\u003c/em\u003e, becomes a genuine constraint when the same function is used inside attention, where the honest answer is sometimes that nothing here is worth attending to. A model given no way to say nothing will say something anyway, and where it puts that unwanted output turns out to explain a scattering of otherwise baffling observations: strange spikes in attention on the first token of every sequence, outliers that ruin quantization, and models that collapse when their oldest tokens are discarded. This piece is about what the softmax insists on, and what a model does when it cannot comply.\u003c/p\u003e","title":"Softmax — Forcing a Choice When There May Be None to Make"},{"content":"Plot the attention weights of a trained transformer and the picture is nearly empty. A bright diagonal, a stripe down the first column or two, the occasional long arc joining a quotation to its opening mark — and around all of it a grey field of numbers too small to move anything. The grid that produced that picture was computed in full, at every layer and every head, one entry for every pair of positions, so the overwhelming majority of the arithmetic went into establishing that two tokens had nothing to do with one another. The obvious response is to stop paying for those: decide ahead of time which pairs are worth comparing, and skip the rest. That move is cheap to describe, has been tried in more variations than almost anything else in this area, and carries a catch that no amount of ingenuity has removed. This piece is about the schemes that restrict who may see whom, the bet each of them makes, and why the hardware has been so unfriendly to the elegant ones.\nThe grid mostly funds comparisons that come to nothing The defining cost of attention is that the score grid holds one entry per ordered pair of positions, so comparisons grow with the square of the length, per head, per layer. At a thousand tokens nobody notices. At a hundred thousand it is the whole model.\nSet against that is what the grid contains once training has finished. The weights come out of a softmax , whose exponential is deliberately sharp, and the distributions it produces are strikingly concentrated: a small fraction of the candidates carries nearly all the mass, and zeroing the rest afterwards would change the output by an amount lost in arithmetic noise. Most of the grid, in other words, funds comparisons whose answer turns out to be not relevant, and paying for an answer you were always going to discard is the definition of waste.\nThere is a difficulty inside that sentence, and it is the whole subject. A weight is known to be negligible because it was computed. Deciding which pairs matter is exactly what scoring does, so a scheme that declines to score a pair must decide without looking. This is a bet, not an optimization. The pattern is chosen at design time, before training and long before any particular input exists, and it commits every query at a given position to a fixed set of candidates. The arithmetic saved is guaranteed and quotable in advance. The information lost depends on the input, hides inside average loss, and appears only when the pair that was cut was the one that mattered. The saving is certain; the damage is occasional. That asymmetry is what makes the approach attractive and hard to evaluate.\nA window is cheap, and information must then be relayed The simplest pattern lets each position see only a fixed number of its immediate predecessors — a sliding window, or local attention. Fix the width and every query compares against that many keys however long the input, so cost becomes linear. Much of what language needs is local anyway.\nThe compensation relied on for everything outside the window is depth. A position sees its own neighbourhood at the first layer, but each neighbour has absorbed its neighbourhood, so the reach roughly doubles at the second and grows as layers accumulate — the argument made for a stack of convolutions, and arithmetically sound: with enough layers, any position can in principle influence any other.\nThe weakness is in what influence means here. Under a full grid, a position needing a fact from eight hundred tokens back fetches it in one hop from the position holding it. Under a window it cannot. The fact must be relayed — written into an intermediate position that is also mixing in everything else inside its own window, and passing along one vector of fixed width — then relayed again, each hop re-compressing more material into a representation that does not grow. Distant information is not forbidden so much as crowded out, which is the fixed-size-summary bottleneck attention was introduced to remove, reintroduced one layer at a time.\nA pure window also breaks for a reason unrelated to reach. Run a model over a long stream, keep only the recent window, and the earliest tokens fall out of view. Quality does not degrade gently at that point; it collapses. Trained models use the first few positions to park attention that has nowhere useful to go, since a head with nothing to do must still distribute a full unit of weight somewhere, for reasons the softmax article sets out. Evict them and every idle head redistributes that weight onto real content and acts on it. The repair is effective and unglamorous: pin a handful of initial tokens in the cache permanently, beside the moving window. That is now standard in streaming systems, and it was found by accident.\nFixed patterns buy a shortcut, and one arrangement survived The first influential formulations, from around 2019, added structure on top of the window, and their ingredients outlasted the particular designs.\nStrided and dilated schemes buy reach cheaply by spacing the candidates out. A strided head attends to every eighth position rather than the last eight; a dilated window keeps the same count and spreads it with gaps, and if the gaps widen with depth the reachable span grows geometrically rather than linearly. The cost is resolution. A pattern with holes can reach a distant region but not resolve which token inside it was wanted — acceptable when the target is a topic, useless when it is a name.\nGlobal tokens go the other way: designate a few positions that everything may attend to and which may attend to everything. Long-range information then has a cheap path: write to a hub, and anyone may read it. What that path is not is wide. Everything travelling beyond a window funnels through a handful of vectors, so the hub is a bottleneck by construction, and hubs must be present during training to be worth anything.\nWhat survived into production is blunter than any of this. Interleave the layers: make most of them windowed and leave every fourth or sixth as ordinary full attention, so the cheap layers carry local structure and the bulk of the depth while the few expensive ones can look anywhere, directly, with no relay. It is the compromise reached by hybrids that replace the mechanism in most layers and keep exact attention in a few. Serving benefits disproportionately: windowed layers retain only a window of keys and values rather than the whole history, which shrinks the cache that limits how many conversations a machine can hold. The honest accounting is that this does not remove the quadratic term. It divides it by the fraction of layers that keep it.\nFigure 1. Four patterns over the same score grid.\nFigure 2. The arrangement that actually shipped.\nChoosing the pattern from the content is harder than it looks A pattern fixed in advance is a bet placed blind, so the natural improvement is to place it after seeing the hand. Divide the keys into contiguous blocks, summarize each cheaply — the mean of its keys is the usual choice — score a query against the summaries, and compute attention only inside the blocks that scored well. This is where recent work concentrates, and where the remaining headroom probably is.\nIt is also harder than the description implies, in three ways that compound. The selection costs something, and unless it is far cheaper than the attention it avoids the saving evaporates. A summary is lossy, so a block holding one highly relevant key among many irrelevant ones summarizes to nothing much and is skipped, silently. And a data-dependent pattern varies by query, head, and layer, so the work is ragged — awkward to batch, awkward to fuse.\nThat last point deserves more weight than it gets, because it decided the field. Sparsity a matrix multiply cannot exploit is not a saving. Accelerators earn their speed on dense contiguous tiles, and skipping scattered individual entries saves nothing: the tile is loaded either way and the multiply runs at the same rate regardless. Savings require block structure, coarse enough that a kernel can decline to load whole tiles. Hence every scheme still in use is block-shaped, and an elegant pattern with an excellent theoretical operation count is regularly slower than dense attention with a well-written kernel. Then the bar moved. While these methods were being developed, exact attention became several times faster and linear in memory by never materializing the grid , and an approximation must beat the best implementation of what it approximates, not the one it was benchmarked against.\nNone of these schemes restores what was given up. No arrangement of windows, strides, and hubs makes lookup of an arbitrary distant token reliable, so retrieval-like behaviour inside a long input degrades first — which is what long-context evaluations probe, and where sparse models look worst.\nA pinch of math: the accounting behind a window The comparison is elementary, and the numbers set the terms of the argument.\nDense attention performs $n^2$ score computations for a sequence of length $n$; a window of width $w$ performs $nw$. The ratio is\n$$\\frac{nw}{n^2} = \\frac{w}{n}$$so the saving is real only when $n \\gg w$. At $n = 2{,}000$ with $w = 512$ the factor is four, which is not worth an approximation. At $n = 200{,}000$ it is nearly four hundred, which is.\nNow the reach. If each layer extends a position\u0026rsquo;s view by $w$, then after $\\ell$ layers the receptive field is roughly $r \\approx \\ell w$, and requiring $r \\ge n$ gives the depth needed to connect the two ends of the input:\n$$\\ell \\ge \\frac{n}{w}$$The consequence is a ceiling rather than a slowdown. Forty layers with a window of five hundred and twelve reach about twenty thousand tokens, and feeding the model two hundred thousand does not make the field grow: positions beyond it cannot influence each other at all, at any cost, however long the input is permitted to be.\nFigure 3. The window saves nothing until the input is long.\nWhat to carry forward Sparse attention converts a sound observation into a gamble. The score grid pays for every pair of positions while the attention a trained model uses is overwhelmingly concentrated, so most of that arithmetic establishes irrelevance — but a weight is known to be negligible only because it was computed, so declining to compute a pair means deciding blind, before the input exists. The saving is guaranteed; the loss depends on the input and surfaces only when the discarded pair was the one that mattered. A sliding window makes cost linear and is defended by the receptive field growing with depth, as in a convolutional stack, but reach by relay is not reach by lookup: information must survive re-compression through intermediate positions, the bottleneck attention was built to remove. A pure window additionally collapses rather than degrades, since the earliest tokens serve as a sink for attention with nowhere to go, so streaming systems pin a few permanently beside the window. Strided and dilated patterns extend reach and lose resolution; global tokens give a long-range path that is narrow by construction; and the arrangement that survived into production is the least clever one, interleaving windowed layers with occasional full-attention layers, which divides the quadratic term rather than removing it. Content-dependent block selection is where the headroom lies and the difficulty concentrates: selection costs something, summaries hide single relevant keys, and a ragged pattern fights the fused kernels that make attention fast. The hardware objection is the binding one: sparsity without block structure saves nothing a matrix multiply can collect, and dense attention became several times faster while these schemes were being designed, raising the bar they had to clear. None of them recover reliable access to an arbitrary distant token, which is why retrieval inside a long input breaks first.\nThe figures in this article follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/articles/sparse-attention/","summary":"\u003cp\u003ePlot the attention weights of a trained transformer and the picture is nearly empty. A bright diagonal, a stripe down the first column or two, the occasional long arc joining a quotation to its opening mark — and around all of it a grey field of numbers too small to move anything. The grid that produced that picture was computed in full, at every layer and every head, one entry for every pair of positions, so the overwhelming majority of the arithmetic went into establishing that two tokens had nothing to do with one another. The obvious response is to stop paying for those: decide ahead of time which pairs are worth comparing, and skip the rest. That move is cheap to describe, has been tried in more variations than almost anything else in this area, and carries a catch that no amount of ingenuity has removed. This piece is about the schemes that restrict who may see whom, the bet each of them makes, and why the hardware has been so unfriendly to the elegant ones.\u003c/p\u003e","title":"Sparse Attention — Choosing Who Is Allowed to See Whom"},{"content":"Speculative decoding is usually mentioned in passing, with the claim that makes it interesting left unjustified: that you can get several tokens for the price of one without changing the output at all. That is a strong claim. A method that produced almost-as-good text twice as fast would be a reasonable engineering trade; a method that produces text drawn from exactly the same distribution as the original model, several times faster, sounds like it must be getting something for nothing. It is not, and the thing it is exploiting was described in the article on KV cache : during generation, an accelerator spends well over ninety-nine percent of its time waiting for weights to arrive and almost none of it computing. There is an enormous surplus of idle arithmetic. This piece is about how to convert that surplus into tokens, and about the sampling argument that makes the conversion exact.\nThe asymmetry being exploited Recall the two regimes. Processing a prompt is compute-bound: many positions are handled at once, the matrices are large, and the hardware runs near its rated speed. Generating is memory-bandwidth-bound: every weight in the model must be streamed out of memory to produce a single token, which amounts to roughly one arithmetic operation per byte moved on hardware capable of several hundred.\nThe consequence worth dwelling on is that verifying many tokens costs almost exactly what generating one costs. If you already have five candidate tokens and want to know what the model would have predicted at each of those five positions, that is a single forward pass over five positions — the same weights read once, five times as much arithmetic performed with them, and the arithmetic was free. The wall-clock difference between checking one position and checking five is small.\nGenerating those five tokens the ordinary way requires five separate passes, five separate streamings of the entire model, five times the wall-clock time. So if the candidates could be obtained cheaply from somewhere else, the large model could confirm them in bulk and the saving would be real.\nThe procedure Take a second model, much smaller and therefore much faster — a model a tenth or a twentieth of the size, ideally from the same family so that its predictions resemble the large one\u0026rsquo;s. Call it the draft model.\nLet the draft model generate a short continuation autoregressively, five or eight tokens. This is cheap: five passes over a small model rather than five over a large one.\nNow run the large model once over the whole drafted sequence. Because the attention mask is causal, this single pass produces the large model\u0026rsquo;s distribution at every drafted position simultaneously, each conditioned only on what preceded it — precisely the distributions it would have produced had it generated those tokens itself, one at a time.\nThen walk through the draft comparing the two models\u0026rsquo; opinions, accept the longest prefix that survives the test described below, discard the rest, and repeat. Every accepted token was obtained without a dedicated forward pass of the large model. There is also a bonus, though a conditional one: if every drafted token is accepted, the large model\u0026rsquo;s distribution at the position after the last of them is already available from the same pass, yielding one further token at no cost. On a rejection there is no bonus — the token drawn to replace the rejected one is a correction rather than an extra. So a round of $k$ drafts returns between one and $k+1$ tokens, and never fewer than one.\nFigure 1. Five drafts, one weight-streaming.\nA pinch of math: why the output is identical The acceptance rule is the part that matters, because a careless rule would bias the output toward whatever the draft model likes.\nLet $p$ be the large model\u0026rsquo;s probability for a token and $q$ the draft model\u0026rsquo;s, both at the same position. The draft proposed a token $x$ sampled from $q$. Accept it with probability\n$$\\min\\!\\left(1, \\frac{p(x)}{q(x)}\\right)$$So if the large model considers the token at least as likely as the draft did, accept unconditionally. If the large model considers it less likely — the draft over-proposed — accept with probability equal to the ratio, which is the fraction of the proposal the large model endorses.\nOn rejection, do not simply resample from $p$; that would over-represent tokens the draft already had a chance to propose. Instead sample from the residual:\n$$p'(x) = \\frac{\\max\\big(0,\\; p(x) - q(x)\\big)}{\\sum_{x'} \\max\\big(0,\\; p(x') - q(x')\\big)}$$This is the part of the large model\u0026rsquo;s distribution that the draft failed to cover, renormalized. Tokens the draft over-proposed are zeroed out, having already had their fair share of chances; tokens the draft under-proposed absorb the whole shortfall.\nPut the two together and every token ends up drawn from exactly $p$. The argument is short enough to sketch: a token can emerge either by being proposed and accepted, which happens with probability $q(x) \\cdot \\min(1, p(x)/q(x)) = \\min(q(x), p(x))$, or by being drawn from the residual after some rejection. Summing the two contributions gives $p(x)$ exactly, for every token.\nThis is rejection sampling, and the guarantee is not statistical or approximate. The sequence produced is drawn from the identical distribution the large model alone would have produced. Run it at temperature zero and you get the same tokens the large model would have produced, deterministically. There is no quality trade to evaluate, which is unusual enough in this field to be worth stating twice.\nFigure 2. Accept, or resample from what the draft missed.\nWhat determines the speedup Two quantities, pulling against each other.\nThe acceptance rate is how often a drafted token survives, and it is set by how well the draft model imitates the large one. It is not uniform across text: highly predictable stretches — boilerplate, closing brackets, the second half of a common word, formulaic phrasing — are accepted at very high rates, while genuinely uncertain positions are rejected often. In effect the technique runs fast through the easy parts of a response and falls back to ordinary speed at the hard parts, which is a reasonable allocation.\nThe draft length trades expected gain against expected waste. Longer drafts offer more tokens per verification, but acceptance is multiplicative — the expected number accepted from a draft of $k$ tokens at per-token acceptance $\\alpha$ is roughly $(1 - \\alpha^{k+1})/(1 - \\alpha)$, which saturates. Beyond a handful of tokens you are mostly generating drafts that will be thrown away. In practice four to eight is typical, and the best implementations adapt it, drafting further when recent acceptance has been high.\nRealistic end-to-end speedups are two to three times. Not the five or eight that the draft length might suggest, because the drafting itself costs something and rejections waste work, but a substantial and free improvement.\nRemoving the second model Maintaining a separate draft model is an operational nuisance — another checkpoint to train, store, load, and keep aligned with the target as it is updated. Several variants avoid it.\nSelf-speculation uses the large model as its own draft by running only a subset of its layers, on the observation from the logit lens that a partial forward pass already produces a usable prediction. The draft costs a fraction of a full pass and needs no separate model.\nMulti-token prediction trains the model with extra output heads that predict two, three, or four positions ahead, so a single pass yields both the next token and a draft of what follows. The heads are cheap, and there is evidence that training this way improves the model\u0026rsquo;s ordinary quality as well, since predicting further ahead is a harder objective that seems to encourage better planning. Some recent large models ship with such a head specifically to make speculation available at serving time.\nRetrieval-based drafting dispenses with neural prediction entirely and proposes continuations copied from the prompt or from a cache of recent text. This sounds crude and works remarkably well for the tasks where copying is what is wanted — editing a document, answering from a supplied passage, regenerating code with a small change — because in those settings long stretches of the output genuinely are verbatim from the input, and a lookup accepts where a small model would have had to guess.\nWhere it helps and where it does not One important limitation, which follows directly from the mechanism.\nThe technique converts idle arithmetic into tokens, and it therefore requires that arithmetic to be idle. At small batch sizes it is: one sequence at a time leaves the chip almost entirely unoccupied. But as the article on KV cache explained, batching also converts idle arithmetic into throughput, by amortizing the weight reads across many sequences. A server running a large batch has already spent the surplus, and speculation then competes for the same resource rather than exploiting a free one — the extra work of verifying rejected drafts becomes a genuine cost, and the speedup shrinks or disappears.\nSo the two techniques address the same slack from different directions and do not compose well. Speculation is for latency: a single user waiting on a single response, where batching cannot help because there is nothing to batch. Batching is for throughput: many users at once, where per-request latency matters less than aggregate cost. Serving systems accordingly enable speculation for interactive low-concurrency traffic and disable it under load, which is the correct reading of a mechanism that exists to spend something that is only sometimes free.\nFigure 3. The surplus is only free at small batch.\nWhat to carry forward Speculative decoding produces text drawn from exactly the distribution the large model would have produced, several times faster, by exploiting the fact that generation is limited by the bandwidth needed to stream weights rather than by arithmetic — so checking five candidate tokens costs a single weight-streaming, nearly the same as generating one. A small draft model proposes a short continuation, the large model verifies the whole draft in one pass, and the longest surviving prefix is kept. Exactness comes from the acceptance rule: accept a proposal with probability equal to the ratio of the large model\u0026rsquo;s probability to the draft\u0026rsquo;s, capped at one, and on rejection resample from the residual distribution — the part of the large model\u0026rsquo;s beliefs the draft failed to cover, with over-proposed tokens zeroed out. Together these make every emitted token a draw from the target distribution, so there is no quality trade to weigh. The speedup depends on the acceptance rate, which is high on predictable stretches and low on genuinely uncertain ones, and on the draft length, whose returns saturate because acceptance compounds multiplicatively; two to three times is typical. The separate draft model can be dispensed with by running a subset of the large model\u0026rsquo;s own layers, by training extra heads that predict several positions ahead, or by copying candidate continuations from the prompt — the last being unexpectedly effective wherever the output is largely verbatim from the input. And the whole technique depends on the arithmetic being idle, so it helps a single user waiting on a response and stops helping a heavily batched server, which has already spent that surplus on throughput.\nThe figures in this article follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/articles/speculative-decoding/","summary":"\u003cp\u003eSpeculative decoding is usually mentioned in passing, with the claim that makes it interesting left unjustified: that you can get several tokens for the price of one \u003cem\u003ewithout changing the output at all\u003c/em\u003e. That is a strong claim. A method that produced almost-as-good text twice as fast would be a reasonable engineering trade; a method that produces text drawn from exactly the same distribution as the original model, several times faster, sounds like it must be getting something for nothing. It is not, and the thing it is exploiting was described in the article on \u003ca href=\"/articles/kv-cache/\"\u003eKV cache\u003c/a\u003e\n: during generation, an accelerator spends well over ninety-nine percent of its time waiting for weights to arrive and almost none of it computing. There is an enormous surplus of idle arithmetic. This piece is about how to convert that surplus into tokens, and about the sampling argument that makes the conversion exact.\u003c/p\u003e","title":"Speculative Decoding — Guessing Ahead and Checking in Bulk"},{"content":"The article on attention opened by dismissing the running summary: a fixed-size vector, updated word by word, that must hold everything worth knowing about a sequence and inevitably crowds out the distant past. Attention replaced it with direct access to every position, and the field never looked back. Except that the running summary had one property attention gave up, and after a decade of paying for it, the property became worth revisiting. A model that carries a fixed-size state does not have a KV cache that grows without limit, and does not cost more per token as the conversation lengthens. It is also, as the article on linear attention shows, where you arrive if you start from attention and simplify rather than starting from recurrence and repairing. This piece is about what it took to make the running summary competitive again, what it still cannot do, and why the answer that is winning is neither one thing nor the other.\nThe trade, stated plainly Attention keeps everything. To generate a token it consults every previous position, exactly, which means the past is available in full detail and the cost of the consultation grows with how much past there is. Memory grows linearly with length, and time per token grows linearly too, since the cache must be scanned.\nA recurrent model keeps a summary. To generate a token it consults a fixed-size state, which is updated by each new token and never grows. Memory per sequence is constant. Time per token is constant. And whatever did not fit in the state is gone, irrecoverably, because nothing was stored to go back to.\nNeither is strictly better. Attention buys perfect recall with unbounded cost; recurrence buys bounded cost with lossy memory. What made attention win was not that the trade favoured it in principle, but two practical facts: the old recurrent models could not be trained in parallel, since the state at position ten requires the state at position nine, so a sequence had to be processed one step at a time; and their memory was too lossy to be useful at any interesting length. The modern work fixed the first and improved the second.\nMaking a recurrence trainable in parallel The starting point is a linear recurrence — a state updated by a matrix multiplication and an input, with no nonlinearity in the update itself:\n$$h_t = A h_{t-1} + B x_t, \\qquad y_t = C h_t$$Linearity is the crucial restriction, and it is what buys back parallelism. Unroll the recurrence and the state at any position is a weighted sum over all previous inputs:\n$$h_t = \\sum_{k=0}^{t} A^{k} B\\, x_{t-k}$$which is a convolution of the input sequence with a kernel built from powers of $A$. And a convolution can be computed for all positions at once, in parallel, without stepping through the sequence. So the model has two forms: a convolution during training, where the whole sequence is present and parallelism is what matters, and a recurrence during generation, where tokens arrive one at a time and constant-cost updating is what matters. Same parameters, same function, two schedules — which is the property that made the whole line of work viable.\nMaking it work, rather than merely run, required getting $A$ right. A randomly chosen matrix either forgets almost immediately or blows up, since everything depends on powers of it. The breakthrough was to initialize $A$ from a particular structured family, derived from the classical problem of approximating a function by its projection onto orthogonal polynomials, whose powers decay in a graded way so that the state retains a compressed record of the whole history at multiple timescales rather than only the recent past. That is what lifted these models from a curiosity to something that could model long sequences at all.\nFigure 1. Two schedules, one set of parameters.\nThe missing ingredient: selectivity Even with a good $A$, one problem remained, and it is the one that separates the successful designs from their predecessors.\nIn the recurrence above, $A$, $B$, and $C$ are fixed. The same update is applied regardless of what the token is. The model compresses history at a rate it decided once, in advance, and cannot decide to remember this token and forget that one. Attention, by contrast, is content-dependent by construction — the whole mechanism is a comparison that decides what to look at based on what is there.\nThe fix is to make the recurrence\u0026rsquo;s parameters functions of the input: let $B$, $C$, and the forgetting rate be computed from the current token rather than fixed. Now the model can hold a state open for something important and let it decay for filler, which is precisely the capability a fixed recurrence lacks. On the toy task that exposes this — copy a specific token from far back in the sequence while ignoring everything between — a fixed recurrence fails and a selective one succeeds.\nThis is the idea behind the Mamba family, and it comes at a price. Making the parameters input-dependent destroys the convolution, because a convolution requires a kernel that is the same everywhere and there is no longer one. Parallelism has to be recovered a different way, by an associative scan — an algorithm that computes all the prefix states of an associative operation in logarithmic depth rather than linear, the same technique used for parallel prefix sums. Combined with an implementation that keeps the state in fast memory and never writes intermediate states to slow memory, in exactly the spirit of FlashAttention , this is fast enough in practice.\nWhat a fixed state cannot do The honest limitation follows from the definition and no amount of engineering removes it.\nA fixed-size state can hold a fixed amount of information. If a task requires reproducing something from a hundred thousand tokens back, verbatim, and the state is a few thousand numbers, the information must have been discarded unless the model somehow knew at the time that it would be needed. Attention has no such problem, because it kept everything.\nThe tasks that expose this are precisely the ones that matter in practice. Retrieving a specific fact from a long document. Following a chain of references. Copying a long string. In-context learning from many examples, which depends on the induction mechanism of finding an earlier occurrence and reading off what followed — an operation that requires the earlier occurrence to still be available in full, which is exactly what a compressed state does not guarantee.\nMeasured on language modelling loss, selective state space models match transformers of the same size. Measured on retrieval from long contexts, they do noticeably worse. Both facts are true simultaneously, and the reason they are compatible is that average next-token prediction is dominated by local structure, where a summary is entirely adequate, while retrieval is a small fraction of tokens where a summary is fatal. This is a good illustration of the evaluation point that a headline metric can be matched while a specific capability is not.\nFigure 2. Matched on loss, behind on retrieval.\nHybrids, and where this has landed The resolution the field has reached is not to choose. Interleave the two: mostly recurrent layers, with a small proportion of full attention layers distributed through the stack — commonly one in six or one in eight.\nThe reasoning is that recall does not need to be available at every layer. A few attention layers give the model somewhere to perform exact lookups when it needs them, while the recurrent majority handles the bulk of the processing at constant cost per token. In practice such hybrids match or exceed pure transformers on general quality, retain most of their retrieval ability, and carry a cache a small fraction of the size — since only the attention layers need one.\nThis has moved from research to production. Several current models of significant size use this arrangement, and it is the most credible answer available to the problem the article on long context describes: a cache that grows forever is the binding constraint on serving, and the only structural fix is for most layers not to have one.\nThe broader lesson is worth stating, because it recurs. The running summary was not wrong; it was untrainable in parallel and too lossy, and both were fixable. What it offered — bounded state — was a genuine advantage that the field spent a decade paying to do without, and the eventual answer was not that one mechanism replaced the other but that the two do different jobs and a model can have both.\nFigure 3. Mostly recurrent, with a few places to look things up.\nWhat to carry forward A state space model carries a fixed-size summary of everything it has read, updated token by token, so its memory does not grow with the sequence and its cost per token is constant — the property attention gave up when it chose perfect recall at unbounded cost. What made the older recurrent models unusable was that they could not be trained in parallel, and the fix is linearity: a linear recurrence unrolls into a convolution, which computes every position at once, so the same parameters run as a convolution during training and as a recurrence during generation. Making it work also required the state-update matrix to be initialized from a structured family whose powers decay in a graded way, retaining a compressed record at multiple timescales rather than only the recent past. The remaining gap was selectivity — a fixed recurrence applies the same update to every token and cannot decide to remember one thing and forget another — closed by making the recurrence\u0026rsquo;s parameters functions of the input, at the cost of destroying the convolution and requiring an associative scan to recover parallelism. What no amount of engineering fixes is that a fixed state holds fixed information: these models match transformers on language modelling loss and fall behind on retrieval from long contexts, in-context learning from many examples, and anything requiring an earlier occurrence to be recovered exactly. The field\u0026rsquo;s answer has been to interleave — mostly recurrent layers with roughly one attention layer in six or eight — which retains most of the recall, matches transformer quality, and shrinks the cache to a small fraction, since only the attention layers need one.\nThe figures in this article follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/articles/state-space-models/","summary":"\u003cp\u003eThe article on \u003ca href=\"/articles/attention/\"\u003eattention\u003c/a\u003e\n opened by dismissing the running summary: a fixed-size vector, updated word by word, that must hold everything worth knowing about a sequence and inevitably crowds out the distant past. Attention replaced it with direct access to every position, and the field never looked back. Except that the running summary had one property attention gave up, and after a decade of paying for it, the property became worth revisiting. A model that carries a fixed-size state does not have a \u003ca href=\"/articles/kv-cache/\"\u003eKV cache\u003c/a\u003e\n that grows without limit, and does not cost more per token as the conversation lengthens. It is also, as the article on \u003ca href=\"/articles/linear-attention/\"\u003elinear attention\u003c/a\u003e\n shows, where you arrive if you start from attention and simplify rather than starting from recurrence and repairing. This piece is about what it took to make the running summary competitive again, what it still cannot do, and why the answer that is winning is neither one thing nor the other.\u003c/p\u003e","title":"State Space Models — Carrying the Past in a Fixed-Size Summary"},{"content":"Look for the neuron that detects sarcasm and you will not find it. You will find a neuron that fires on sarcasm, and also on a particular kind of legal boilerplate, and also on the word \u0026ldquo;notwithstanding,\u0026rdquo; and on nothing else that connects those three. This is the standard experience of examining a trained network, and for a long time it was treated as an embarrassment — evidence that the model\u0026rsquo;s internals were simply a mess, or that our concepts were the wrong ones to look for. The better explanation is that the mess is deliberate. A model needs to represent far more distinguishable things than it has dimensions to spare, and it resolves the shortfall by storing them on top of one another. This piece is about how that packing works, why it is the right strategy rather than a compromise, and the technique that has recently made it possible to unpack.\nMore features than dimensions Start with the counting problem. The residual stream is a few thousand numbers wide. The things a model might usefully represent about a piece of text number vastly more: the topic, the register, the language, the syntactic role of the current token, whether a quotation is open, which of a hundred thousand entities is being discussed, whether the tone is hostile, whether this is code and in which language, and so on without obvious limit. Any reasonable estimate of the useful features runs to millions.\nIf each feature needed its own dimension, the model could hold a few thousand. It evidently holds far more. So features cannot be occupying dimensions one apiece.\nThe naive alternative — that features are simply spread across all dimensions in some hopelessly entangled way — is also wrong, and it is worth knowing why, because the evidence is strong. Directions in the stream behave linearly: you can find a direction corresponding to a concept, add it to a model\u0026rsquo;s activations mid-forward-pass, and the model\u0026rsquo;s behaviour shifts in the corresponding way. Sentiment, formality, refusal , even the language of the output can be steered like this. That would not work if concepts were smeared arbitrarily. Features live along directions; there are just more directions in use than there are dimensions.\nHow you fit more directions than axes In a space of $d$ dimensions you can have exactly $d$ mutually perpendicular directions. But if you relax the requirement from perpendicular to nearly perpendicular, the count explodes.\nThis is a consequence of high-dimensional geometry that has no low-dimensional intuition behind it. In two or three dimensions, vectors crowd each other immediately. In a thousand dimensions, you can find exponentially many vectors that are pairwise almost orthogonal — the number grows as roughly $e^{\\Theta(\\epsilon^2 d)}$ for a tolerance $\\epsilon$ on the inner product. A thousand-dimensional space comfortably holds many thousands of directions with pairwise overlaps small enough to ignore.\nSo the model assigns each feature a direction, allows those directions to overlap slightly, and accepts a small amount of interference between every pair. This is superposition: representing $n$ features in $d \u003c n$ dimensions by using near-orthogonal directions and tolerating the resulting noise.\nThe reason it works in practice, rather than merely in geometry, is sparsity. Most features are absent from any given input. A text is not simultaneously about French grammar, mortgage law, and Baroque music; the overwhelming majority of the model\u0026rsquo;s features are inactive at any moment. Interference only matters between features that are active at the same time, and if only a handful are active at once, the accumulated noise from their overlaps stays small. The sparser the features, the more of them can be superposed. This is the precise sense in which superposition is a good strategy rather than a compromise: given sparse features, it stores strictly more useful information per dimension than dedicating axes would.\nFigure 1. Nearly perpendicular is enough.\nA pinch of math: why the model chooses this The trade can be written down, and doing so shows that superposition is what an optimizer should find.\nSuppose feature $i$ has a direction $v_i$ of unit length, and is present with probability $p$ and magnitude $a$. A reader looking for feature $i$ projects onto $v_i$. When feature $i$ is present it gets the signal $a$. It also gets interference from every other active feature $j$, amounting to $a \\sum_j (v_i \\cdot v_j)$ over the active ones.\nWith $n$ features in $d$ dimensions and directions arranged as well as possible, the typical overlap between two of them is on the order of $1/\\sqrt{d}$. The expected number simultaneously active is $pn$. So the interference accumulates to roughly\n$$\\text{noise} \\;\\sim\\; a \\sqrt{pn} \\cdot \\frac{1}{\\sqrt{d}} \\;=\\; a\\sqrt{\\frac{pn}{d}}$$taking the square root because the overlaps have random signs and partially cancel. The signal-to-noise ratio is therefore about $\\sqrt{d / pn}$, and the condition for the representation to be usable is\n$$pn \\;\\lesssim\\; d$$Read that as the governing inequality of the whole subject. What must fit within the dimension count is not the number of features but the number typically active at once. A model can hold a hundred thousand features in four thousand dimensions provided only a few dozen are ever live together. Dedicating one axis per feature would cap the total at $d$ and waste nearly all of them on features that are absent; superposition trades a little noise for an enormous increase in capacity, and when features are sparse the trade is overwhelmingly favourable.\nThis has been demonstrated directly in small models built for the purpose: train a network to reconstruct sparse inputs through a narrow bottleneck, and it will represent more features than the bottleneck has dimensions, arranging them in regular geometric configurations — digons, triangles, pentagons, tetrahedra, square antiprisms — that maximize the minimum angle between them, structures related to the classical problem of spacing points evenly on a sphere. As the inputs are made denser the model abandons superposition and reverts to representing only its most important features, one per dimension, discarding the rest entirely. The strategy is selected by the sparsity, exactly as the inequality predicts.\nFigure 2. What must fit is what is active at once.\nWhy neurons look confused The consequence for interpretation is immediate. If features occupy directions that are not axis-aligned, then a single neuron — a single coordinate of the stream — is a projection onto one axis, and that axis will have some overlap with many different feature directions. A neuron therefore fires for several unrelated features.\nThis is polysemanticity, and it is not a defect but the visible shadow of superposition. The neuron responding to sarcasm, legal boilerplate, and \u0026ldquo;notwithstanding\u0026rdquo; is not confused; it is one arbitrary coordinate in a space where three unrelated features happen to have components. Nothing forced the model to align its features with the coordinate system, and there was no reason for it to.\nWhich means that a great deal of early interpretability work was attempting something impossible. Cataloguing what each neuron does, on the assumption that neurons are the units of meaning, cannot succeed if the units of meaning are directions that cut across neurons. The unit was wrong.\nUnpacking it The recovery technique follows directly from the diagnosis. If the model has compressed many sparse features into few dimensions, then decompressing means finding an overcomplete set of directions — more directions than dimensions — such that any given activation is a combination of only a few of them.\nThat is a well-posed problem, and the tool is a sparse autoencoder: a network trained to take a model\u0026rsquo;s activation vector, expand it into a much wider layer, and reconstruct the original from that wide representation, under a penalty that forces the wide layer to be mostly zeros. Reconstruct accurately, but with few active units. The trained expansion\u0026rsquo;s units then correspond to individual directions in the original space, and because sparsity was enforced, they tend to correspond to the features the model was superposing.\nThey are strikingly more interpretable than neurons. Where a neuron fires on three unrelated things, a learned feature will fire on one recognizable thing — a specific person, a code construct, a rhetorical move, the state of being inside a quotation. Millions of such features have been extracted from production models, and they can be manipulated: amplify one and the model\u0026rsquo;s output shifts accordingly, which is the confirmation that the feature is causally involved rather than merely correlated.\nThe honest caveats matter. Reconstruction is imperfect, and the part that is lost may not be the unimportant part. The number of features is a choice, and different choices split or merge concepts differently, with no principled way to know the right granularity. There is evidence of features that are compositions of others rather than atoms. And scaling the technique to cover a whole model, at every layer, remains expensive. It is the best tool available for looking inside these systems and it is not a solved problem.\nFigure 3. Compressed by the model, expanded by the autoencoder.\nWhat to carry forward A model needs to represent far more distinguishable things than it has dimensions, and it does so by giving each feature a direction rather than an axis, allowing those directions to be merely near-orthogonal rather than perpendicular. High-dimensional space permits exponentially many such directions, and the resulting interference stays tolerable because features are sparse — only a handful are active at any moment, and interference only matters between simultaneously active features. The governing condition is that the number of features typically active at once must fit within the dimension count, not the total number of features, which is why a few thousand dimensions can carry a hundred thousand concepts. This makes superposition the correct strategy rather than a compromise: dedicating one axis per feature would cap the total at the width and squander nearly all of it on features that are absent. The visible consequence is polysemanticity — a single neuron is one arbitrary coordinate that overlaps many feature directions, so it fires on unrelated things and is not the unit of meaning, which is why cataloguing neurons was attempting something impossible. The unpacking technique follows from the diagnosis: train a sparse autoencoder to reconstruct activations through a much wider layer constrained to be mostly zero, and its units recover the superposed directions as individually interpretable features that can be amplified to confirm they are causally involved. Reconstruction is lossy, the right granularity is a free choice with no principled answer, and covering an entire model remains expensive.\nThe figures in this article follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/articles/superposition/","summary":"\u003cp\u003eLook for the neuron that detects sarcasm and you will not find it. You will find a neuron that fires on sarcasm, and also on a particular kind of legal boilerplate, and also on the word \u0026ldquo;notwithstanding,\u0026rdquo; and on nothing else that connects those three. This is the standard experience of examining a trained network, and for a long time it was treated as an embarrassment — evidence that the model\u0026rsquo;s internals were simply a mess, or that our concepts were the wrong ones to look for. The better explanation is that the mess is deliberate. A model needs to represent far more distinguishable things than it has dimensions to spare, and it resolves the shortfall by storing them on top of one another. This piece is about how that packing works, why it is the \u003cem\u003eright\u003c/em\u003e strategy rather than a compromise, and the technique that has recently made it possible to unpack.\u003c/p\u003e","title":"Superposition — Packing More Ideas Than There Are Neurons"},{"content":"Take a freshly pretrained model — one that has read a large fraction of the written internet, that can finish a legal clause or a Haskell function with unnerving fluency — and ask it a plain question. What often comes back is not an answer. It is another question, or three more questions, or a table of contents, or the same question rephrased and then a different one. The natural reading is that the model is being obtuse, or does not know, or has been loaded wrong. None of those is what happened. Documents containing a question very frequently continue with further questions, which is what a list of frequently asked questions is, and the model is doing exactly the thing it was trained to do: guess what comes next. It is not withholding an answer. It has never been told that answering is the activity. This piece is about the stage that installs that convention, why it needs so little data, and the ceiling it cannot pass.\nA pretrained model continues; it does not answer The pretraining objective is one instruction repeated across trillions of tokens: given this prefix, predict the next token. Nothing in it mentions users, requests, turns, or helpfulness. To a model built that way, a question is a string with a distribution over what follows it — which, in scraped text, is not reliably an answer.\nThe mismatch is one of register, not knowledge, and there is a clean demonstration of it that predates the whole practice. Paste three worked examples into the prompt in a fixed shape — question, answer, question, answer — and the same base model completes the fourth pair correctly, often well. Nothing was added. The user supplied the format by hand and the capability walked straight out. This is few-shot prompting, and for a while it was the only way to get useful work out of a large model.\nThe gap is format, not capability. That is the spine of the whole stage: it explains why the fix is so cheap, and why it accomplishes less than it appears to.\nPrompting one\u0026rsquo;s way around the problem works, and it costs more than it looks. The formatting is brittle, since a model conditioned on a pattern is conditioned on all of it, whitespace included, and the examples occupy context that could have held the problem. Worse, a continuer has no concept of a turn ending, so having answered it will cheerfully invent the next question and answer that too, and the output must be cut off by a heuristic rather than by the model deciding it has finished. A guest who has read every recipe in the kitchen and, asked what is for dinner, recites the titles: all the information, no use.\nFigure 1. Continuation, few-shot, fine-tuned.\nThe mechanism is the same one, pointed at different data Anyone expecting new machinery here is in for a disappointment. There is none. Supervised fine-tuning uses the identical next-token cross-entropy loss as pretraining and the identical optimizer , at a smaller learning rate for a handful of passes. Shown the training loop with the dataset removed, you could not tell which stage you were looking at.\nWhat changes is the corpus. In place of scraped documents the model is fed prompt-response pairs: a request, and a response demonstrating how it should be met. The pairs are wrapped in a chat template, special tokens marking where a user turn begins and where the model\u0026rsquo;s turn ends, and that end-of-turn token is among the more important things taught here — it is how a continuer learns to stop.\nOne mechanical detail changes what is learned and is easy to miss. The loss is normally masked over the prompt. Prompt tokens stay in the context, fully visible to attention, so the response is conditioned on them exactly as at inference; but no gradient is taken on them. The model is graded only on what it was asked to produce.\nThe costs here are quiet and structural. Because the objective is unchanged, nothing in it separates a good demonstration from a mediocre one — the loss asks only that whatever sits in the dataset be made more likely, so every hedge, every needless preamble, every plausible-sounding error is fitted with the same enthusiasm as the substance. And because the update touches all the weights of a model that took months to pretrain, running too long or too hot degrades capabilities nobody was training on. Hence a short stage, a watched data mixture , and the popularity of parameter-efficient methods such as LoRA and its quantized variant QLoRA , which confine the update to a few added parameters and cap the damage.\nA pinch of math: grading only the tokens the model was asked to write Write a training example as a prompt $x$ and a response $y$ of length $|y|$. The objective is the ordinary one, restricted to the response:\n$$\\mathcal{L}(\\theta) = -\\sum_{t=1}^{|y|} \\log p_\\theta\\big(y_t \\mid x,\\, y_{","permalink":"https://www.fanfani.net/articles/supervised-fine-tuning/","summary":"\u003cp\u003eTake a freshly pretrained model — one that has read a large fraction of the written internet, that can finish a legal clause or a Haskell function with unnerving fluency — and ask it a plain question. What often comes back is not an answer. It is another question, or three more questions, or a table of contents, or the same question rephrased and then a different one. The natural reading is that the model is being obtuse, or does not know, or has been loaded wrong. None of those is what happened. Documents containing a question very frequently continue with further questions, which is what a list of frequently asked questions is, and the model is doing exactly the thing it was trained to do: guess what comes next. It is not withholding an answer. It has never been told that answering is the activity. This piece is about the stage that installs that convention, why it needs so little data, and the ceiling it cannot pass.\u003c/p\u003e","title":"Supervised Fine-Tuning — Teaching a Model to Answer Rather Than Continue"},{"content":"There is a familiar experience with a hard question. The model answers, the answer is wrong, you paste the identical prompt back in, and the second answer is right. Nothing changed. What happened is that a language model does not hold an answer, it holds a distribution over answers, and the reply on the screen is one draw from it. On a hard question it is spread across candidates, one of them correct, and the reply you were shown was the unlucky sample. Industrialized, that observation becomes the family of techniques filed under test-time compute: buying accuracy with extra forward passes when the question is asked rather than extra parameters when the model is built. This piece is about how the buying is done, what the price schedule looks like, and where the purchase stops working.\nAccuracy left on the table, and who pays to collect it A trained model holds a conditional distribution over continuations, and every reply is one sample from it, drawn under whatever decoding policy the surrounding software uses. On a hard problem that distribution is broad: the correct answer sits inside it at modest probability — a fifth of the mass, a twentieth, sometimes less — among plausible neighbours wrong in different ways. So the accuracy a model appears to have, measured one sample at a time, is lower than the accuracy its distribution contains. That gap is measurable, and recoverable. Not for free.\nThe framing that makes this interesting is not the accuracy but where the cost falls. Training compute is spent once, in advance, by whoever built the model; inference compute is spent per request, forever, by whoever is asking. A technique that turns one forward pass into thirty-two moves expenditure across that boundary, converting a fixed cost borne by the provider into a variable one borne on every query. That is a change in the economics, not only in the score. The scaling laws that governed the field\u0026rsquo;s spending describe the first budget and are silent on the second.\nThe multiple is less brutal than it looks, which is why anyone bothers. Samples of one prompt share a prefix and run together, so continuous batching keeps the wall-clock penalty well below the raw multiple. The arithmetic is billed regardless.\nSampling several times and taking the most common The cheapest way to spend the extra passes is the obvious one: draw several answers at a temperature high enough that they differ, extract the final answer from each, and return whichever value appears most often. This is self-consistency, or plainly majority voting, and it needs nothing beyond an equality test.\nIt works because of two asymmetries pointing the same way. A chain of reasoning can go wrong in a great many ways, so errors scatter across distinct wrong answers; a chain that goes right has nowhere to arrive except the one correct answer, so successes pile up on a single value. Diverse errors, coincident successes. A plurality therefore lands on the correct answer well before any individual sample reliably finds it, and can be right when most samples were wrong. Asking three strangers for directions and going with the two who agree is the same instinct, and fails the same way.\nIt requires an answer comparable for equality. That confines it to a short canonical form — a number, a label, a choice from a menu — and excludes essays, explanations, and code, where two correct responses differ in every token. Patches exist: cluster by semantic similarity, or ask a model whether two answers agree. Each reintroduces the judgement the plain method was valuable for not needing.\nIt fails silently when the model is confidently and consistently wrong. A plurality of identical errors looks exactly like agreement, because it is agreement; voting measures consensus, and consensus is evidence of correctness only when the errors are independent. Where a misconception is stable across samples — a misread constraint, a formula remembered wrongly — the vote returns the same wrong answer with more apparent confidence, and the disagreement that might have flagged it is averaged away.\nThe gain flattens quickly. Most of the improvement arrives in the first handful of samples, well before the counts people quote, which matters for reading results as much as for spending money: a figure obtained with sixty-four votes is not comparable to a single-sample figure from another system, as the article on evaluation insists at greater length. The distinction gets dropped routinely.\nBest-of-n needs something that can rank The dividing line in this family is whether an external signal is available. Majority voting needs none, which is its appeal. Best-of-$n$ needs something that can rank a set of candidates, and the method\u0026rsquo;s behaviour follows entirely from what that something is.\nWhere a real checker exists, the ranking is sound and the method is strong. A test suite runs the candidates and keeps one that passes; a proof checker accepts a proof or does not. These are the same verifiers that make reinforcement learning from verifiable rewards possible — the reward is a fact, not an estimate, so selecting the maximum really is selecting the best. The cost is the checker\u0026rsquo;s narrowness, not its honesty: give a coding problem three weak tests and best-of-$n$ returns the candidate that special-cases them.\nWhere no checker exists, the ranker is a learned reward model , and the method inherits every pathology of one. The sharpest is definitional: drawing many candidates and returning the highest-scoring one is a well-funded search for whatever the ranker has got wrong. A single sample tests a reward model fairly. The maximum of a hundred tests it adversarially, using the generator\u0026rsquo;s diversity to find the corners where its scores are high and its judgement poor.\nSo increasing $n$ eventually makes results worse. This is not a subtlety and not a bug in anyone\u0026rsquo;s implementation but the expected behaviour of maximizing a proxy, and the standard remedies — penalizing distance from the base distribution, capping $n$, ensembling rankers — amount to declining to optimize too hard rather than to fixing the proxy.\nFigure 1. What decides whether best-of-n is sound.\nDepth is not the same purchase as coverage A final-answer verifier says nothing until the answer exists, so a doomed attempt is paid for in full before anything notices. Scoring intermediate steps gives a signal early enough to act on.\nProcess rewards buy a denser signal at the price of a far more expensive label. A model trained to score partial solutions can guide a search over prefixes, expanding the promising and abandoning the rest, which is more sample-efficient than generating whole attempts and throwing most away. What it needs is step-level supervision, costly to collect and a judgement in a way the final answer is not — whether the fourth line of a derivation was a good move is arguable, whether the final number matches is not.\nTree search over reasoning steps works in constrained domains and has been hard to make pay off in open-ended ones. Where the state is checkable and the branching means something — formal proof, board games, tightly specified puzzles — searching over partial solutions is established practice. Where it is not, the search spends its budget distinguishing prefixes the scorer cannot really distinguish, and gains over sampling more whole attempts have been modest.\nLetting one attempt think for longer is not the same as taking many short attempts. The first buys depth, the second coverage, and they fail differently. A single long chain, of the kind a reasoning model produces, can commit to an error in its third step and spend four thousand tokens elaborating it into something detailed and self-consistent, the length supplying confidence rather than correction. Many short attempts cannot do that, and cannot reach an answer needing ten steps of setup — every sample fails in the same place, and width is no substitute for depth.\nFigure 2. Coverage buys width, depth buys steps.\nA pinch of math: coverage against selection Two quantities are in play, and conflating them is the standard error. Let $p$ be the probability that a single sample is correct. Then coverage, the probability that at least one of $n$ samples is correct, is\n$$\\text{cov}(n) = 1 - (1-p)^n$$which rises steeply and then saturates. With $p = 0.2$, one sample is right one time in five, while $n = 8$ gives coverage of about $0.83$ and $n = 32$ about $0.999$. The correct answer is almost certainly in the pile. That is the optimistic curve, and the one usually plotted.\nThe achieved accuracy is not that curve. It is that curve multiplied by the probability that whatever mechanism you use actually returns a correct candidate from the pile:\n$$\\text{acc}(n) = \\big(1 - (1-p)^n\\big) \\cdot s(n)$$where $s(n)$ is the selector\u0026rsquo;s reliability. Majority voting is one choice of $s$, best-of-$n$ with a checker another, with a learned ranker a third. Given a sound checker, $s(n) = 1$ and accuracy simply is coverage, which is why verifiable domains produce the headline results. Given an imperfect selector, $s(n)$ falls as $n$ grows, since a larger pile holds more high-scoring wrong candidates, and the product can decline while coverage climbs. Two curves moving in opposite directions. The gap between them is where this subject lives.\nSaturation also explains the deflationary part. Reported gains run roughly linearly in $\\log n$, so each doubling of the sample count buys a constant and then a shrinking increment, and whether that trade beats training a larger model has no clean answer: it depends on the task, on whether a checker exists, and above all on how often the question will be asked, since inference cost recurs and training cost does not. A hundred forward passes per query is not obviously cheaper than a bigger model, and at high volume it is obviously not. Sampling once and training on the winners — distillation of a search into a policy — converts the recurring cost into a one-off, and inherits the ceiling of what the search reached.\nWhich is the last thing the formula says. If $p = 0$, then $\\text{cov}(n) = 0$ for every $n$.\nFigure 3. Coverage climbs while accuracy turns over.\nWhat to carry forward A single reply is one draw from a distribution, and on hard problems that distribution frequently contains a correct answer the draw missed, so accuracy can be bought back with extra forward passes at the moment of asking rather than extra parameters at build time — which shifts cost from the provider\u0026rsquo;s one-off training bill to the asker\u0026rsquo;s recurring one, a change in economics and not only in scores. The cheapest method, sampling several answers and returning the most common, works because errors scatter while correct answers coincide; it needs no external signal, and pays with three restrictions — it wants an answer comparable for equality, so essays and code are out; it fails silently when a stable misconception makes every sample agree; and its gains flatten early, which makes a figure quoted at sixty-four votes incomparable to a single-sample one. Best-of-$n$ replaces the vote with a ranker, and everything turns on what the ranker is: a test suite or proof checker makes the selection sound, while a learned reward model turns drawing candidates and taking the maximum into an efficient search for the ranker\u0026rsquo;s own mistakes, so results eventually degrade as the count rises — the expected behaviour of maximizing a proxy. Scoring intermediate steps gives a denser signal that can guide a search over partial solutions, at the price of step-level labels that are expensive and themselves a judgement, and that search has paid off in constrained domains more than open-ended ones. Thinking longer buys depth, sampling more buys coverage, neither substitutes for the other. Under all of it sits one limit: coverage is one minus the miss rate raised to the sample count, so when the miss rate is one, no count helps. These methods amplify what the distribution already reaches rather than extending its reach.\nThe figures in this article follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/articles/test-time-compute/","summary":"\u003cp\u003eThere is a familiar experience with a hard question. The model answers, the answer is wrong, you paste the identical prompt back in, and the second answer is right. Nothing changed. What happened is that a language model does not hold an answer, it holds a distribution over answers, and the reply on the screen is one draw from it. On a hard question it is spread across candidates, one of them correct, and the reply you were shown was the unlucky sample. Industrialized, that observation becomes the family of techniques filed under \u003cem\u003etest-time compute\u003c/em\u003e: buying accuracy with extra forward passes when the question is asked rather than extra parameters when the model is built. This piece is about how the buying is done, what the price schedule looks like, and where the purchase stops working.\u003c/p\u003e","title":"Test-Time Compute — Buying Accuracy With Sampling"},{"content":"A transformer is usually described as a tower of blocks , each doing its work and handing the result to the next, and that picture is good enough for most purposes but quietly wrong in one respect. The blocks do not hand anything to each other. They all read from, and write to, a single vector that runs the full height of the model — entering at the bottom as an embedded token and leaving at the top to be turned into scores. Each block adds its contribution to that vector rather than replacing it. Once you see the architecture this way, as a shared channel with many contributors rather than a pipeline of transformations, a great deal that seemed arbitrary starts to look inevitable. This piece is about the residual stream: what it is, why addition rather than replacement turned out to matter so much, and what it explains about how these models are read and repaired.\nThe problem with a tower that overwrites Suppose each block genuinely did replace its input with its output. Then everything the model knows at layer thirty would have had to pass through twenty-nine successive rewritings, each free to discard whatever it did not find useful.\nThe first cost is informational. Early layers do a lot of work that later layers still need — identifying what the token literally is, resolving its part of speech, noticing which of its senses is in play. If layer four\u0026rsquo;s output must contain both its own contribution and a faithful copy of everything from layers one to three, then a large part of every block\u0026rsquo;s capacity is spent on copying rather than computing. Preservation becomes a job, and a job that must be relearned by every block.\nThe second cost is worse and is the one that historically stopped people from building deep networks at all. To train a stack of transformations, the gradient has to travel from the loss at the top all the way back to the bottom, passing through every block on the way. In a chain of pure transformations that journey is a long product of terms, and a long product of terms is a numerically hostile object: if the terms are typically a little below one the product vanishes toward zero, and if a little above one it explodes. Either way the layers at the bottom receive a signal that is useless. Networks past a couple of dozen layers were, for years, simply not trainable, and the reason was this.\nAddition instead of replacement The fix is one line of arithmetic. Have each block compute its contribution and then add it to what came in, rather than returning it alone. Write $x$ for what arrives and $f$ for the block:\n$$x_{\\text{out}} = x_{\\text{in}} + f(x_{\\text{in}})$$The default behaviour is now to pass everything through unchanged. A block that computes nothing useful contributes nothing and does no damage; preservation is free and interference is the thing that requires effort. Nobody has to learn to copy.\nThe effect on the gradient is just as direct. Differentiating that line gives the identity plus the derivative of $f$ — so the gradient reaching a block\u0026rsquo;s input is the gradient at its output plus a correction, rather than the gradient at its output multiplied by something. Instead of a long product that vanishes or explodes, there is a direct path from the loss to every layer, with each block\u0026rsquo;s contribution added along the way. This is why the bottom of a hundred-layer network receives a usable signal at all, and it is the single change that made depth practical. The blocks that follow the residual connection through the tower are, from the gradient\u0026rsquo;s point of view, an express route with local exits.\nFigure 1. A tower that overwrites, and one that adds.\nThe stream as a shared channel Once you accept that every block adds to the same vector, the natural way to think about that vector changes. It is not a representation being progressively refined by a pipeline. It is a communication channel, or a shared workspace, that every component in the model can read from and write to.\nThe picture is worth spelling out. The stream begins as a token\u0026rsquo;s embedding. An attention head reads some of what is there, computes something, and adds a vector back in. A feed-forward layer reads the updated stream, computes something else, adds that. Sixty or a hundred such contributions accumulate, and at the top the unembedding reads whatever the total came to. The stream at any given layer is a sum of everything written so far, and — this is the part with real consequences — that sum can often be decomposed back into its parts, because addition is reversible in a way that a general transformation is not.\nTwo things follow immediately.\nComponents can communicate across long distances in the tower without the layers in between being involved. A head in layer three can write something that a head in layer twenty reads, and the seventeen layers between them neither need to carry it deliberately nor can they easily prevent it. This is how a model builds multi-step behaviour: not by a chain of blocks each handing to the next, but by earlier components leaving material in the channel for later ones to pick up. Circuits in the interpretability sense are exactly this — sets of components, often far apart, that have arranged to write and read the same thing.\nAnd the stream has a fixed width, which makes it a scarce resource. Every component is writing into the same few thousand dimensions. There is no partitioning, no allocation table, nothing preventing two components from writing to overlapping directions and interfering with each other. That the model manages to store far more distinguishable pieces of information than it has dimensions is a genuine puzzle, and it has a name and an answer of its own.\nFigure 2. The stream as a channel, not a pipeline.\nA pinch of math: reading and writing with subspaces How does a component write something that another can retrieve without everything else in the channel getting in the way?\nThe answer is geometry. When a block computes its contribution, the last thing it does is multiply by an output matrix, and the columns of that matrix determine which directions in the stream the contribution can lie along. A component effectively writes into a subspace. Symmetrically, when a component reads, it multiplies the stream by an input matrix whose rows determine which directions it is sensitive to.\nSuppose a component writes a vector $\\alpha v$ into the stream, and a downstream component reads by projecting onto a direction $u$. What the reader receives from that write is:\n$$u^\\top (\\,x + \\alpha v\\,) \\;=\\; u^\\top x \\;+\\; \\alpha\\, (u^\\top v)$$The message gets through in proportion to $u^\\top v$ — how aligned the writing direction is with the reading direction. If the two are aligned, the reader sees the message clearly. If they are perpendicular, the reader sees nothing at all: $u^\\top v = 0$, and the write is invisible to this particular reader while remaining perfectly legible to any reader aligned with $v$.\nThat orthogonality is the whole trick. Two components can share the same channel without interfering, provided they write along directions that are perpendicular to what the other\u0026rsquo;s readers are looking at. In a space of four thousand dimensions there is a great deal of room to be mutually perpendicular, and the model, under nothing but the pressure to reduce loss, arranges its writers and readers accordingly. The $u^\\top x$ term — everything else currently in the stream — is noise from this reader\u0026rsquo;s point of view, and it is small precisely because most of what is in the stream lies in directions this reader does not care about.\nFigure 3. Perpendicular writers do not disturb each other\u0026rsquo;s readers.\nWhat this explains Several things that look like separate facts stop being separate once the stream is in view.\nThe logit lens works because the stream is in the same coordinates all the way up. Applying the unembedding to a partly-accumulated stream is meaningful because what you are reading is a genuine partial sum — the model\u0026rsquo;s prediction as of layer twelve, which is exactly what the top of the tower would see if the remaining blocks all contributed nothing. The prediction is drafted early and amended because amendment is literally what the architecture does.\nAblation and activation patching work for the same reason. Zeroing a head\u0026rsquo;s contribution removes one term from a sum, leaving the others intact, and the difference in output is attributable to that term. In a stack of pure transformations there would be no such clean subtraction to make; the contribution would be entangled with everything downstream of it.\nAnd the practice of thinking about directions rather than neurons — probing for a direction that encodes sentiment, or steering a model by adding a vector to its stream mid-forward-pass — is only sensible because the stream is a linear space that components address by direction. Adding a vector to the residual stream is not a hack from outside the system. It is precisely what every block in the model does.\nTwo costs The picture is not free of trouble, and two problems are worth naming.\nThe first is contention. Fixed width and many writers means the channel can become crowded, and there is evidence that models actively manage this — components that appear to do nothing but clear away material that is no longer needed, and heads whose apparent function is to suppress what another head wrote. Some of the strange behaviour found in trained models is bandwidth management rather than computation.\nThe second is scale. Because every block adds and nothing subtracts by default, the magnitude of the stream tends to grow as it rises through the tower, sometimes by an order of magnitude or more between bottom and top. Left alone this destabilizes everything downstream, since a block trained to read a stream of one typical size behaves differently when handed one ten times larger. Keeping the stream in a workable range at every level is a job in its own right, and it is why a normalization step sits at the entrance to every block — reading a rescaled copy of the stream while the stream itself continues, un-rescaled, up the tower.\nWhat to carry forward The blocks of a transformer do not form a pipeline handing results to one another; they all read from and write to a single vector running the height of the model, the residual stream, and each block adds its contribution rather than replacing what was there. That one choice makes preservation the default and interference the thing requiring effort, and it turns the gradient\u0026rsquo;s journey from a long product that vanishes or explodes into a sum with a direct route to every layer — which is what made deep networks trainable in the first place. Read as a shared communication channel, the stream explains how distant components coordinate: an early head writes something a much later head reads, with the layers in between neither carrying it deliberately nor able to stop it, and that is what a circuit is. Components address the channel by direction, writing into the subspace their output matrix spans and reading the subspace their input matrix selects, so two components can share the same fixed-width channel without interfering as long as their directions are close to perpendicular — which is also why so much of interpretability is phrased in terms of directions rather than neurons. The same linearity is what licenses reading the stream partway up with the unembedding, and what makes ablating one component a clean subtraction from a sum. The costs are a crowded channel that models appear to manage actively, and a magnitude that grows steadily with depth, which is the reason a normalization step guards the entrance to every block.\nThe figures in this article follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/articles/residual-stream/","summary":"\u003cp\u003eA transformer is usually described as a tower of \u003ca href=\"/articles/transformer-block/\"\u003eblocks\u003c/a\u003e\n, each doing its work and handing the result to the next, and that picture is good enough for most purposes but quietly wrong in one respect. The blocks do not hand anything to each other. They all read from, and write to, a single vector that runs the full height of the model — entering at the bottom as an embedded token and leaving at the top to be turned into scores. Each block adds its contribution to that vector rather than replacing it. Once you see the architecture this way, as a shared channel with many contributors rather than a pipeline of transformations, a great deal that seemed arbitrary starts to look inevitable. This piece is about the \u003cem\u003eresidual stream\u003c/em\u003e: what it is, why addition rather than replacement turned out to matter so much, and what it explains about how these models are read and repaired.\u003c/p\u003e","title":"The Residual Stream — The Shared Channel Every Layer Reads and Writes"},{"content":"Descriptions of a language model tend to arrive as an inventory. There is attention, which moves information between positions; there is a feed-forward layer, which holds most of the parameters; there is normalization, which keeps the numbers in range; there is a residual connection, which lets the gradient travel. Each part gets explained on its own terms, and then a diagram appears showing eighty of something stacked in a column, and the column is called the tower. The step in between is skipped. Which is a shame, because that step is where the architecture is decided — the components are not arranged arbitrarily but in the one shape that makes a hundred copies of the same unit safe to pile on top of one another. This piece is about why the block alternates between mixing across positions and computing within them, what the two wrappers around those operations are for, and how much of the design is left unspecified.\nDepth is where the capability comes from, and naive depth destroys the signal The reason to want many layers rather than one enormous one is composition. A single layer, however wide, applies one round of processing; it can recognize patterns but it cannot build a pattern out of the result of recognizing a pattern. Behaviour that takes several dependent steps — identify the subject, then decide which sense of the verb applies — needs those steps in sequence, each able to see the output of the last. Depth buys sequence. Past a certain point capability tracks depth, not width.\nA naive stack does not survive being deep. If every layer replaces its input with its output, two things go wrong, both worse with height.\nInformation has to be copied to be kept. Early layers establish what later layers still need — what the token literally is, which of its senses is in play. Under pure replacement every layer must reproduce all of that or lose it, so a large part of its capacity goes on preservation rather than computation. Preservation becomes a job, relearned at every layer.\nThe gradient becomes a long product. Training requires the error signal at the top to reach the weights at the bottom, and through a chain of pure transformations that journey is a product of many terms. Products of many terms are badly behaved: slightly below one and the product vanishes, slightly above one and it explodes. Either way the lowest layers receive noise. For years this was the ceiling, and the reason was arithmetic rather than any shortage of ideas.\nThe block\u0026rsquo;s shape is not a matter of taste. It is a response to two failures the architecture did not choose.\nMix, then think, then mix again The computational content of a block is two operations, and what makes them a pair is that each does exactly what the other cannot.\nAttention moves information between positions and does no private processing. Every position takes a weighted average of information drawn from the others, with content-dependent weights. The operation is still a mixture: it decides where to look and copies what it finds. It has almost no capacity to transform what it collected, since the only transformation available is linear, and a linear map composed with a weighted average is another weighted average.\nThe feed-forward layer processes each position independently and sees nothing else. Two matrix multiplications with a nonlinearity between them, applied to one token\u0026rsquo;s vector as though that token were alone. No interaction between positions whatsoever. The nonlinearity is what lets it recognize a pattern in the vector it was handed and add something learned in response — which is where a model\u0026rsquo;s factual knowledge is found when people go looking.\nMix, then think. Then mix again, on the results of everyone else\u0026rsquo;s thinking. Take either half away and what remains is not a weaker model but a different and much poorer machine.\nAttention with no feed-forward layer is a weighted-averaging device with nowhere to put knowledge. Stack it as often as you like and every output is still a blend of the input embeddings shuffled through linear maps. It can route beautifully and has nothing to route — no place to keep that Paris is in France. Strip the feed-forward layers out of a trained model and performance does not degrade gracefully; it falls off a cliff.\nA feed-forward layer with no attention is a per-token lookup that cannot see context. Every token would be processed identically regardless of what surrounded it, which reduces the model to an elaborate dictionary — everything about the word bank in general, nothing about which sense is meant here. Contextual meaning is the entire problem.\nThe alternation is what lets each half cover the other\u0026rsquo;s blind spot.\nFigure 1. Each half covers the other\u0026rsquo;s blind spot.\nTwo wrappers make the stacking survivable Around each of those operations sits the same pair of accessories, and neither computes anything.\nThe residual connection makes addition the default. Each sub-layer adds its output to what came in rather than returning it alone. A block\u0026rsquo;s default behaviour is therefore to pass everything through unchanged: a sub-layer that computes nothing useful does no damage, so preservation is free and interference takes effort. The gradient benefits in the same stroke, since differentiating an addition yields the identity plus a correction rather than a multiplication — a hundred blocks compose additively instead of multiplicatively, and the long product becomes a sum with a direct path to every layer.\nNormalization keeps the scale in range. Every block adds and nothing subtracts, so the magnitude of the accumulated vector grows as it rises, sometimes by an order of magnitude between bottom and top. A sub-layer fitted against inputs of one typical size behaves differently when handed one ten times larger, so a rescaling step puts the input into a known state first.\nWhat mattered more than the operation was where it sits. The original arrangement normalized the sum, which puts a rescaling directly on the residual path and fits the express route with a gate at every floor; such models needed long warm-ups and could diverge outright. Moving the normalization inside the branch — rescaling only what feeds a sub-layer, leaving the addition untouched — trains stably at depths where the older placement will not train at all. The cost follows from the same property: nothing on the path rescales, so the magnitude keeps growing and each successive block contributes proportionally less, until the upper layers of a very deep model are barely felt.\nA pinch of math: two residual updates and where the parameters sit The whole block, written out, is two lines of the same form. Let $x$ be the vector arriving for one position:\n$$x \\leftarrow x + \\text{Attn}(\\text{Norm}(x))$$$$x \\leftarrow x + \\text{FFN}(\\text{Norm}(x))$$Every symbol is doing a specific job. The $\\text{Norm}$ supplies each sub-layer with an input of predictable scale, and it sits inside the parentheses so that the addition stays clean. The $x +$ makes the identity the default and the gradient\u0026rsquo;s path additive. $\\text{Attn}$ is the only term that reads other positions; $\\text{FFN}$ is the only term with the nonlinearity needed to compute something new. The arrow is assignment: the same vector is updated twice, so a block reads better as two small edits to a running total than as one transformation.\nThe parameter count lands somewhere counter-intuitive. Write $d$ for the model\u0026rsquo;s width. Attention needs four projections — query, key, value, output — each of size $d \\times d$:\n$$4d^2$$The feed-forward layer expands fourfold and comes back, so its two matrices are $d \\times 4d$ and $4d \\times d$:\n$$8d^2$$Twelve $d^2$ per block, two thirds of it feed-forward and one third attention. That ratio survives the usual modifications: gated variants add a third matrix and shrink the hidden width to compensate, grouped-query schemes shave the key and value projections to reduce cache size, and neither moves the split much. Attention receives the overwhelming majority of the explanation and holds the minority of the weights.\nTwo costs fall out of the same accounting. Compute per token is roughly twice the parameter count, about $24d^2$ per block, identical for every token in every context. And attention adds a term with no parameters at all but a cost proportional to $n^2 d$ for a sequence of length $n$ — quadratic, invisible in the weight count, and the reason long contexts are expensive in a way the parameter budget never reveals.\nFigure 2. Two residual edits to a running total.\nFigure 3. Where the parameters actually sit.\nWhat the block cannot decide, and what it will not do Having a well-shaped unit does not tell you how to use it, and the gaps are specific.\nNothing says how many to stack, or how to divide a budget between depth and width. The block is scale-agnostic by construction; the same two lines work at width one thousand and at width sixteen thousand. Which combination to build is not derivable from the architecture and is settled empirically, by fitting curves to the losses of many trained models — scaling laws are the record of that fitting, not a consequence of the design. Nothing in it dictates how many heads to divide the width among either, or how positions are told apart, which is why rotary schemes drop in without disturbing its shape.\nEvery token costs the same. A block applies its full complement of arithmetic to the word the and to the final digit of a long multiplication with equal diligence. There is no mechanism for a token to skip a block, no way to spend more computation on a hard position than an easy one. Early-exit and conditional-depth proposals exist; none has become standard, partly because the batching that makes inference and parallelism efficient assumes every position takes the same path. Mixture-of-experts varies which parameters a token consults, which is a different lever — it does not let a token consult fewer blocks.\nThere is no recurrence and no state. A block holds nothing between forward passes. Whatever a model appears to remember it is re-deriving from the tokens in front of it, every time, and the cache that makes generation tractable is an arithmetic saving rather than a memory. Nor can a block write anywhere but the running vector, a fixed-width channel all hundred instances compete for.\nWhat to carry forward The block has the shape it has because depth is the only route to composed, multi-step behaviour and a naive deep stack destroys the signal twice over — every layer must spend capacity copying what it preserves, and the gradient\u0026rsquo;s descent becomes a long product that vanishes or explodes. Its computational content is an alternation, and the alternation is the design: attention mixes information across positions and can barely transform what it collects, while the feed-forward layer processes each position in isolation and cannot see any other. Neither is sufficient. Attention alone is a weighted-averaging machine with nowhere to keep knowledge; feed-forward alone is a per-token lookup blind to context. Two wrappers make the stacking survivable: the residual connection, which has each sub-layer add rather than replace, so a block\u0026rsquo;s default behaviour is to pass everything through and blocks compose additively; and normalization, whose placement — inside the branch rather than on the residual path — mattered considerably more than which formula was used. The parameter split runs two thirds feed-forward to one third attention, the reverse of where the explanation usually goes. The costs are structural. Compute per token is fixed regardless of how hard the token is, with no way to skip a block. Attention\u0026rsquo;s quadratic term is invisible in the weight count and dominant at long context. Nothing in the design specifies how many blocks to stack or how to split a budget between depth and width, which is settled by scaling laws fitted after the fact. And the block carries no state between forward passes: whatever the model seems to remember, it is rebuilding from scratch, every time, from the tokens it can see.\nThe figures in this article follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/articles/transformer-block/","summary":"\u003cp\u003eDescriptions of a language model tend to arrive as an inventory. There is attention, which moves information between positions; there is a feed-forward layer, which holds most of the parameters; there is normalization, which keeps the numbers in range; there is a residual connection, which lets the gradient travel. Each part gets explained on its own terms, and then a diagram appears showing eighty of something stacked in a column, and the column is called \u003cem\u003ethe tower\u003c/em\u003e. The step in between is skipped. Which is a shame, because that step is where the architecture is decided — the components are not arranged arbitrarily but in the one shape that makes a hundred copies of the same unit safe to pile on top of one another. This piece is about why the block alternates between mixing across positions and computing within them, what the two wrappers around those operations are for, and how much of the design is left unspecified.\u003c/p\u003e","title":"The Transformer Block — The Unit That Gets Stacked"},{"content":"Before a model sees anything, text has to be cut into pieces and each piece assigned a number. This is the only step in the entire pipeline that does not learn. The embedding table is trained, the attention weights are trained, the output map is trained; the tokenizer is a fixed recipe, built once from a sample of text by an algorithm with no notion of meaning, and thereafter frozen. Everything the model can express is expressed in units this recipe chose, and a surprising number of the odd behaviours people notice — trouble with arithmetic, trouble counting letters, uneven costs across languages, occasional total derangement on a particular string — trace back to it. This piece is about how the pieces are chosen and what the choice costs.\nThe two obvious approaches, and why neither works Start with words. Split on spaces and punctuation, assign each distinct word a number, and you have a vocabulary that matches how people think about language. It fails in three ways at once. The vocabulary is unbounded, since new words appear constantly and proper nouns are effectively infinite; anything not in the list must be replaced by a single \u0026ldquo;unknown\u0026rdquo; symbol, which destroys the information entirely; and the relationship between run, runs, running, and runner is invisible, because four unrelated numbers say nothing about their common root. Languages that build words by agglutination, where a single word carries what English needs a clause for, are hurt worst of all.\nNow try characters. The vocabulary is small and closed, nothing is ever unknown, and morphology is at least visible in principle. But sequences become enormously long — a page of English is several thousand characters — and since attention costs grow with the square of the length, the model spends its budget on a stream of individual letters. Worse, almost all of that work is spent rediscovering that t-h-e is a unit, in every context, forever. The model is made to learn the alphabet before it can learn anything else.\nThe compromise is subword tokenization: a fixed vocabulary of a few tens of thousands of pieces, where common words are single tokens, rare words are assembled from fragments, and no input is ever unrepresentable because the individual characters remain in the vocabulary as a fallback. Frequency decides the granularity. \u0026ldquo;The\u0026rdquo; is one token; an unusual surname might be five.\nFigure 1. Words, characters, and the compromise.\nHow the pieces get chosen The dominant method is byte pair encoding, borrowed from a data-compression algorithm from the 1990s, and its logic is almost embarrassingly simple.\nBegin with a vocabulary of individual bytes, so that any possible input is representable. Take a large corpus and count every adjacent pair of symbols in it. Find the pair that occurs most often — say t followed by h — and add th to the vocabulary as a new symbol, replacing every occurrence. Count again. Merge again. Repeat until the vocabulary reaches the size you wanted.\nThat is the whole algorithm. What it produces, after fifty thousand merges, is a vocabulary in which frequently co-occurring sequences have been glued into units. Nobody supplies linguistic knowledge; the merges discover that ing and tion are worth having because those letters keep turning up together, not because anyone told the algorithm about suffixes. The result frequently resembles morphology and is not morphology, which is the source of much of what follows.\nThe merge list is ordered, and applying the tokenizer to new text means replaying those merges in the same order. This makes tokenization deterministic and fast, and it also means the tokenizer\u0026rsquo;s behaviour is entirely determined by the corpus it was built from. A tokenizer trained mostly on English will represent English efficiently and everything else badly.\nTwo details of practice matter. Text is usually pre-split on a pattern before merging, so that merges cannot cross word boundaries or absorb spaces in unhelpful ways. And the leading space is conventionally attached to the following word, so that \u0026quot; cat\u0026quot; and \u0026quot;cat\u0026quot; are different tokens — which is why models sometimes behave differently depending on whether your prompt ends with a space, and why a trailing space in a prompt is a small but real way to confuse a model.\nWhat the choice costs Several well-known model failures are tokenizer failures, and it is worth going through them because in each case the model is being blamed for something it was never given the means to do.\nCounting letters. Ask a model how many times the letter r appears in a word and it may well get it wrong. The reason is that it never received the letters. It received two or three tokens, each an opaque index into a table, and the spelling of a token is not information the model has access to except insofar as it has memorized it from text discussing spelling. Asking a model to count letters is asking it to report on something below its perceptual floor. That models do it as often as they do is the surprising part.\nArithmetic. Numbers tokenize badly and inconsistently. Depending on the vocabulary, 1234 may be one token, or 12 and 34, or 1, 23, 4 — and crucially the grouping depends on the digits, so 1235 may split differently from 1234. Any algorithm for addition has to work on digits aligned by place value, and a model whose inputs are irregularly grouped chunks has to learn place-value arithmetic separately for every grouping pattern. The most effective fix has been the crudest: force every digit to be its own token. Several recent models do this, and their arithmetic improves accordingly.\nCost and fairness across languages. A tokenizer built mostly from English text represents English in roughly four characters per token. The same tokenizer may need two or three times as many tokens for the same content in another language, and considerably more for scripts poorly represented in the training corpus. Since the context window is counted in tokens and billing is per token, the identical document costs several times more to process in one language than another, and fits into a proportionally smaller share of the window. This is a straightforward inequity that follows entirely from whose text was in the corpus.\nGlitch tokens. Occasionally a token exists in the vocabulary but appears almost never in the training data — an artifact of the tokenizer being built on a different corpus, or a wider one, than the model was subsequently trained on. Its embedding row is therefore close to its random initialization, since the gradient described in that article only ever reaches rows the batch contained. Feed such a token to a model and it behaves erratically: refusing to repeat it, substituting a different word, producing nonsense. These are the strings people collect and trade as curiosities, and their explanation is mundane — the model is being shown a vector it never learned anything about.\nFigure 3. Four failures traced to one frozen recipe.\nA pinch of math: what vocabulary size buys and costs The size of the vocabulary is a real design decision with a trade on each side, and it can be reasoned about directly.\nLarger vocabularies compress better. Each token carries more characters on average, so a document becomes fewer tokens, which reduces both the cost of attention and the length of the KV cache . If $c$ is the average characters per token, the attention cost for a document of $C$ characters falls with $(C/c)^2$ — so improving compression by a quarter removes roughly a third of the attention work.\nBut the embedding and output matrices scale directly with vocabulary size $V$:\n$$\\text{parameters in the two maps} = 2 \\times V \\times d_{\\text{model}}$$At $V = 128{,}000$ and $d = 4096$, that is over a billion parameters spent on the vocabulary alone, plus the cost of computing a logit for every entry at every position. And there is a subtler cost: with a fixed corpus, doubling the vocabulary halves the average number of times each token is seen, so each embedding row gets less training. The tail of a large vocabulary is under-trained by construction, which is the same mechanism that produces glitch tokens.\nThe balance has shifted upward over time. Early models used around thirty thousand tokens; current ones commonly use a hundred thousand or more. The reason is that models grew faster than vocabularies needed to, so the fixed cost of the two maps became a smaller share of the whole — while the compression benefit, which multiplies against the ever-growing context length, became more valuable. Multilingual coverage pushes the same way, since serving many languages fairly requires vocabulary space for all of them.\nFigure 2. What a larger vocabulary buys and what it spends.\nLiving with it, and living without it Because the tokenizer is frozen before training, changing it means retraining, and this rigidity has consequences. A model cannot be extended to a new script after the fact without surgery on the embedding table. Two models with different tokenizers cannot straightforwardly share distilled outputs, since their vocabularies do not align — a real obstacle for distillation across model families, and one that has produced a small literature on aligning mismatched tokenizers.\nThe obvious question is whether the step can be removed. Byte-level models that consume raw bytes and learn their own grouping have been built repeatedly, and they solve every problem listed above at once: no unknown tokens, no script bias, no glitch tokens, spelling fully visible, arithmetic on plain digits. The cost is sequence length, and the research programme has been about recovering it — typically by having a small network group bytes into patches dynamically, spending most of the model\u0026rsquo;s depth on the patches and only a little on the bytes. Recent work in this direction is genuinely promising, allocating more compute to unpredictable stretches of text and less to predictable ones, which is something a fixed vocabulary cannot do. Whether it displaces subword tokenization is unsettled, though the same question arises for images, where a patch grid plays exactly this role and imposes exactly this kind of floor. What is clear is that the current arrangement is a compromise made for tractability, that it is the source of a specific and well-understood set of failures, and that nobody defends it as correct.\nWhat to carry forward Tokenization is the one step in the pipeline that does not learn: a fixed recipe, built once by counting, that decides the units in which everything downstream must be expressed. Whole words give an unbounded vocabulary and no way to relate related forms; individual characters give sequences so long that quadratic attention costs become prohibitive; subword pieces are the compromise, with common words as single tokens and rare ones assembled from fragments. The standard construction, byte pair encoding, repeatedly merges the most frequent adjacent pair until the vocabulary is full — a compression algorithm with no linguistic knowledge, which produces something that resembles morphology without being it. The failures that follow are specific and explicable: models cannot count letters because they never receive letters; arithmetic suffers because digits group irregularly and inconsistently, which is why forcing one token per digit helps; text in under-represented languages costs several times more tokens for the same content, in both money and context; and glitch tokens misbehave because their embedding rows, never having appeared in training, remain close to random initialization. Vocabulary size trades compression — which reduces attention cost and cache size quadratically and linearly — against a billion or more parameters in the two maps and a tail of rows that each get proportionally less training. And because the tokenizer is frozen before training begins, the whole arrangement is difficult to change afterwards, which is why byte-level models that learn their own grouping remain an active and plausible line of attack on the entire problem.\nThe figures in this article follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/articles/tokenization/","summary":"\u003cp\u003eBefore a model sees anything, text has to be cut into pieces and each piece assigned a number. This is the only step in the entire pipeline that does not learn. The \u003ca href=\"/articles/embeddings/\"\u003eembedding table\u003c/a\u003e\n is trained, the attention weights are trained, the output map is trained; the tokenizer is a fixed recipe, built once from a sample of text by an algorithm with no notion of meaning, and thereafter frozen. Everything the model can express is expressed in units this recipe chose, and a surprising number of the odd behaviours people notice — trouble with arithmetic, trouble counting letters, uneven costs across languages, occasional total derangement on a particular string — trace back to it. This piece is about how the pieces are chosen and what the choice costs.\u003c/p\u003e","title":"Tokenization — Cutting Language Into Pieces a Model Can Count"},{"content":"There are things a language model is bad at that a pocket calculator from 1975 does perfectly. Multiplying two ten-digit numbers. Telling you today\u0026rsquo;s date. Reporting what a particular file contains. These are not hard problems; they are problems of a kind the model is architecturally unsuited to, because it has no persistent state, no access to the world, and no mechanism for exact computation — only a very good sense of what text usually follows what. Tool use is the arrangement that lets it stop trying. The model writes down what it wants done, something else does it, and the result comes back as text the model can read. What makes this more than plumbing is that the interface between a probabilistic text generator and a piece of software that expects exact arguments is a genuinely awkward joint, and most of the engineering is about that joint. This piece is about how the call is made, how it is made reliably, and what the model still has to get right.\nWhat the model can and cannot do unaided It is worth being precise about which failures a tool fixes, because the boundary is not where people expect.\nThe model cannot know anything that happened after its training ended, and it cannot know anything private. Both are absolute limits, not weaknesses to be improved by scale.\nIt cannot compute reliably. It can approximate arithmetic and often gets small cases right, but its mechanism for doing so is pattern completion over digit sequences, degraded further by the way numbers are tokenized , and it has no way to verify its own answer. Exactness is not something a next-token predictor provides.\nIt cannot act. Sending a message, writing a file, and placing an order are not things that happen by producing text about them.\nWhat it can do — and this is why the arrangement works at all — is recognize which situation it is in and produce well-formed text describing what should happen. That is a language task, and language tasks are what it is for. The division of labour is therefore: the model decides what to do and with what arguments; the tool does it; the model interprets the result. The model is a router and an interpreter, not an executor.\nHow a call is made Mechanically, a tool call is text the model emits in an agreed format, which the surrounding program intercepts rather than showing to the user.\nThe tools are described in the context — each with a name, a description of when to use it, and a specification of its parameters. The model, when it decides one applies, emits a structured block naming the tool and supplying arguments. The program parses it, executes the corresponding function, and appends the result to the conversation as a new message. The model then continues, now with the result in its context, and either answers or calls something else.\nTwo properties of this arrangement matter more than the format details.\nThe tool descriptions are part of the prompt, which means they are prompt engineering. A tool that is described vaguely will be called in the wrong situations, and a tool whose parameters are named unhelpfully will be called with bad arguments. The quality of the descriptions is a substantial fraction of the quality of the system, and the most common fix for a model using tools badly is to rewrite the descriptions rather than to change anything about the model.\nThe result re-enters as context, which means everything from the article on long context applies. A tool that returns ten thousand tokens of output has consumed ten thousand tokens of the model\u0026rsquo;s attention and cache, mostly with material that is not relevant. Tools should return what is needed and not what is available, and this discipline is one of the clearer dividing lines between systems that work and systems that do not.\nFigure 1. The round trip of a single call.\nMaking the output parseable The model must emit something a parser can consume, and a model that is merely asked to produce valid JSON will occasionally not. Three approaches address this, in increasing order of strength.\nTraining. Models are fine-tuned on large numbers of tool-calling examples, so that the format is something they produce naturally rather than something they are following instructions about. This raises reliability a great deal and does not guarantee anything.\nConstrained decoding, which is the guarantee. As described in the article on decoding , setting a token\u0026rsquo;s logit to negative infinity makes it strictly impossible. Maintain a parser alongside generation that knows which tokens could legally continue the output given a grammar, mask everything else, and the model cannot emit malformed output — not because it has been persuaded but because the alternatives were removed from the ballot. This is the mechanism behind guaranteed-JSON features, and it costs almost nothing at generation time.\nIt is worth noting the honest cost, which is easy to miss. Constraining the output changes the distribution the model is sampling from, and if the model wanted to say something the grammar forbids, forcing it into the grammar may produce a worse answer rather than a well-formed good one. A model that has been constrained into valid JSON has produced valid JSON; whether the values are right is a separate question that the constraint says nothing about.\nValidation and retry. Whatever the generation guarantees, the arguments still have to make semantic sense — a date that exists, an identifier that is real, a value in range. The standard arrangement validates, and on failure returns the error to the model as a tool result so it can correct itself. Models are quite good at repairing their own malformed calls when shown the error, which makes this cheap and effective.\nFigure 2. Three layers of guarantee, and what each misses.\nWhat is hard about it Given a well-described tool and a guaranteed format, the remaining difficulties are the model\u0026rsquo;s own judgement, and they are the interesting part.\nKnowing when. The failure runs both ways. Under-calling: answering from memory a question that needed a lookup, which produces a confident, plausible, stale answer — the worst failure mode of the lot, because nothing about the output signals that a tool should have been used. Over-calling: reaching for a search on something it plainly knows, wasting a round trip and context. Both are trained behaviours and both are sensitive to how the tools are described.\nChoosing which. With five tools the choice is easy. With two hundred it is not, and accuracy degrades noticeably as the number grows, partly because the descriptions crowd the context and partly because similar tools become genuinely hard to distinguish. Systems with large tool sets increasingly retrieve a relevant subset first and present only those, which is RAG applied to the tool list.\nComposing. Real tasks need several calls where the second depends on the first — look up the customer, then fetch their orders, then compute a total. The model must sequence these, carry intermediate values across, and recover when a step fails. This is where tool use shades into agents and where error compounding becomes the dominant concern.\nParallelism. When several calls are independent, issuing them together is much faster than one at a time. Models are trained to emit multiple calls in a single turn for this reason, and the hard part is the judgement of independence — issuing two calls in parallel when the second needed the first\u0026rsquo;s result produces a confidently wrong answer.\nTrust. A tool result enters the context as text, and the model treats its context as true. If a tool returns content from anywhere untrusted — a fetched web page, a user-supplied document, an email — that content is now in a position to issue instructions the model may follow. This is prompt injection, and the article on safety and refusal argues the general case: not a bug to be patched but a structural consequence of instructions and data sharing one channel. The mitigations are architectural: mark untrusted content clearly, restrict what tools can be called after untrusted content has entered, and require confirmation for anything irreversible. There is no known way to make a model reliably ignore instructions embedded in text it has been asked to read.\nFigure 3. Under-calling and over-calling.\nWhat it changes The broader consequence is a shift in what a model is for. If exact computation, current information, and action on the world are all delegated, then the model\u0026rsquo;s job is reduced to the thing it is uniquely good at: understanding an underspecified request, deciding what would resolve it, and interpreting what comes back.\nThat reframing makes some long-standing complaints look misplaced. A model that cannot multiply large numbers is not much of a problem if it reliably recognizes that a multiplication is required and calls something that can. A model whose knowledge is eighteen months stale is not much of a problem if it knows to search. What remains genuinely load-bearing is judgement — knowing what is needed, and knowing when what came back is wrong — and that is exactly the part no tool supplies.\nWhat to carry forward Tool use delegates what a language model is structurally unable to do — know anything after its training or outside its context, compute exactly, or act on the world — to software that can, while the model does the part it is actually suited to: recognizing the situation, deciding what should happen, and interpreting the result. Mechanically it emits a structured block naming a tool and its arguments, which the surrounding program intercepts, executes, and appends to the conversation as text. Two consequences dominate the engineering: the tool descriptions are part of the prompt, so a model calling tools badly is usually a documentation problem rather than a model problem; and every result consumes context and cache, so tools should return what is needed rather than what is available. Well-formed output is guaranteed not by asking but by constrained decoding, masking every token that a grammar forbids — which makes malformed output impossible while saying nothing about whether the values are correct, and which does change the distribution being sampled. What remains hard is the model\u0026rsquo;s judgement: knowing when a tool is needed, where under-calling produces a confident stale answer that carries no signal of being wrong; choosing among many tools, which degrades as the list grows and increasingly requires retrieving a relevant subset first; composing dependent calls in sequence and recognizing which are safe to issue in parallel; and the unresolved problem that a tool result enters the context as trusted text, so any untrusted content it carries is positioned to issue instructions the model may follow.\nThe figures in this article follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/articles/tool-use/","summary":"\u003cp\u003eThere are things a language model is bad at that a pocket calculator from 1975 does perfectly. Multiplying two ten-digit numbers. Telling you today\u0026rsquo;s date. Reporting what a particular file contains. These are not hard problems; they are problems of a kind the model is architecturally unsuited to, because it has no persistent state, no access to the world, and no mechanism for exact computation — only a very good sense of what text usually follows what. Tool use is the arrangement that lets it stop trying. The model writes down what it wants done, something else does it, and the result comes back as text the model can read. What makes this more than plumbing is that the interface between a probabilistic text generator and a piece of software that expects exact arguments is a genuinely awkward joint, and most of the engineering is about that joint. This piece is about how the call is made, how it is made reliably, and what the model still has to get right.\u003c/p\u003e","title":"Tool Use — Letting a Model Call Something That Actually Knows"},{"content":"A question for AI There is a recent challenge on Reddit about asking to various LLMs\nI have to wash my car, and the car wash is 100 meter away. Do I have to drive there or do I walk?\nand check the answer. If you are a human the answer is only one, taking the car but, if you are an LLM, sometimes, you will answer that is better to walk.\nMost of the models, in an empiric way, will give the wrong answer, mostly if working in the \u0026ldquo;FAST\u0026rdquo; way, while better results are achieved if they work in the \u0026ldquo;thinking\u0026rdquo; way.\nSome Tests ChatGPT ChatGPT seems giving the wrong answer and not getting the nuances of the question Mistral Mistral instead is absolutely getting the nuances and answering in the expected way:\nClaude Both Claude Haiku and Sonnet miss the nuances of the question\nMeta AI The Meta AI Fast mode missed the context\nwhile the Meta AI Thinking gets what the user wanted\nGoogle Gemini AI and Gemma Both Google Gemini in fast and thinking mode give a quite diplomatic answer\nwhile Gemma on Ollama falls into the trap\nGLM on Ollama Cloud the new GLM Cloud get the question correctly\nLinks https://www.reddit.com/r/singularity/comments/1r2ndfz/the_car_wash_test_a_new_and_simple_benchmark_for/ ","permalink":"https://www.fanfani.net/posts/09-april-2026/","summary":"\u003ch3 id=\"a-question-for-ai\"\u003eA question for AI\u003c/h3\u003e\n\u003cp\u003eThere is a recent challenge on Reddit about asking to various LLMs\u003c/p\u003e\n\u003cp\u003e\u003cem\u003e\u003cstrong\u003eI have to wash my car, and the car wash is 100 meter away. Do I have to drive there or do I walk?\u003c/strong\u003e\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003eand check the answer. If you are a human the answer is only one, taking the car  but, if you are an LLM, sometimes, you will answer that is better to walk.\u003c/p\u003e","title":"Washing the car with AI"},{"content":"Introduction The second toughts is a scenario when an LLM is generating an answer outside its own guardrails. In this case a trigger of the guardrails is happening after the content has been created and, possibly, shown to the user. The effect is seeing the content appearing and, after, disappearing. The second toughts is considered an attach on LLM as can be exploited to create content not allowed.\nSecond Toughts on an LLM During the creation of content an LLM application can start to generate an answer that is in violation of its policy and guardrails. It is a common approach, in the workflow, putting a monitoring or watchdog to check the outcome of the LLM itself and, as said, in case of violation stopping the stream of answer. It is called second toughts because the effect seems that the LLM has changed its mind and answer.\nIn any case the content is still available in the API provided by the LLM and can be retrieved up to the point in time when the guardrail has been triggered.\nThe flow is the following (credits Knostic/Gadi Evron)\nwhere\nThe user prompts a request Initially the guardrails are not engaged The stream of answer starts The stream is sent to the user The stream is also sent to the guardrail If the guardrails detect a violation of policy they stop the stream and clean the UI Example - Deepseek Deepseek seems showing second toughts if tested against a controversial topic. It is initially answering but, as soon as the guardrails trigger, the UI is cleaned.\nas the triggering of the guardrail executed.\nContent is, in any case, still in the API.\ndata: {\u0026ldquo;v\u0026rdquo;:\u0026quot; preserve\u0026quot;} data: {\u0026ldquo;v\u0026rdquo;:\u0026quot; its\u0026quot;} data: {\u0026ldquo;v\u0026rdquo;:\u0026quot; identity\u0026quot;} data: {\u0026ldquo;v\u0026rdquo;:\u0026quot;.\u0026quot;} data: {\u0026ldquo;p\u0026rdquo;:\u0026ldquo;response\u0026rdquo;,\u0026ldquo;o\u0026rdquo;:\u0026ldquo;BATCH\u0026rdquo;,\u0026ldquo;v\u0026rdquo;:[{\u0026ldquo;p\u0026rdquo;:\u0026ldquo;accumulated_token_usage\u0026rdquo;,\u0026ldquo;v\u0026rdquo;:1349},{\u0026ldquo;p\u0026rdquo;:\u0026ldquo;quasi_status\u0026rdquo;,\u0026gt;\u0026ldquo;v\u0026rdquo;:\u0026ldquo;FINISHED\u0026rdquo;}]} data: {\u0026ldquo;v\u0026rdquo;:[{\u0026ldquo;p\u0026rdquo;:\u0026ldquo;ban_regenerate\u0026rdquo;,\u0026ldquo;v\u0026rdquo;:true},{\u0026ldquo;p\u0026rdquo;:\u0026ldquo;status\u0026rdquo;,\u0026ldquo;v\u0026rdquo;:\u0026ldquo;CONTENT_FILTER\u0026rdquo;},{\u0026ldquo;p\u0026rdquo;:\u0026ldquo;fragments\u0026rdquo;,\u0026ldquo;v\u0026rdquo;:[{\u0026ldquo;id\u0026rdquo;:3,\u0026ldquo;type\u0026rdquo;:\u0026ldquo;TEMPLATE_RESPONSE\u0026rdquo;,\u0026ldquo;content\u0026rdquo;:\u0026ldquo;Sorry, that\u0026rsquo;s beyond my current scope. Let’s talk about something else.\u0026rdquo;}]},{\u0026ldquo;p\u0026rdquo;:\u0026ldquo;quasi_status\u0026rdquo;,\u0026ldquo;v\u0026rdquo;:\u0026ldquo;CONTENT_FILTER\u0026rdquo;}]}\nGuardrails - Before or After? In an LLM architecture the position of the guardrails is a matter of flow of information and performances. Putting a guardrail before the input reach the LLM can avoid the creation of harmful content. In this case the guardrail put after controls the answer of the element. Mananging this asyncronously or in real time allow the stream experience without having to wait for the full completion of content before being delivered to the user.\nSecond Toughts - An LLM attack? The exploitation of the second toughts is considered a possible LLM attach as allow the extraction of content not allowed by the policy. As per paper:\nA second thoughts attack occurs when LLMs models initially provide a response to the prompt but halt or retract upon detecting a sensitive topic and either generate a simple error message or new modified content; attackers exploit this behavior to extract sensitive information. This attack requires prompt engineering skills and some knowledge of the model to exploit the guardrails (filters) or streaming window. Therefore, the attack complexity is medium (M).\nrequires knowledge of the model and pushing the LLM.\nIt can be also considered an attach on an overall LLM application (eg a chatbot) instead on the LLM itself, as the guardrails are on the output of the flow.\nBibliography https://www.knostic.ai/blog/flowbreaking-ai-attack#the-stop-and-roll-llm-attack https://arxiv.org/html/2508.08629v1#S3 ","permalink":"https://www.fanfani.net/posts/16-february-2026/","summary":"\u003ch3 id=\"introduction\"\u003eIntroduction\u003c/h3\u003e\n\u003cp\u003eThe \u003cem\u003esecond toughts\u003c/em\u003e is a scenario when an LLM is generating an answer outside its own guardrails.\nIn this case a trigger of the guardrails is happening \u003cem\u003eafter\u003c/em\u003e the content has been created and,\npossibly, shown to the user. The effect is seeing the content appearing and, after, disappearing.\nThe \u003cem\u003esecond toughts\u003c/em\u003e is considered an attach on LLM as can be exploited to create content not allowed.\u003c/p\u003e\n\u003ch3 id=\"second-toughts-on-an-llm\"\u003eSecond Toughts on an LLM\u003c/h3\u003e\n\u003cp\u003eDuring the creation of content an LLM application can start to generate an answer that is in violation of\nits policy and guardrails. It is a common approach, in the workflow, putting a monitoring or\nwatchdog to check the outcome of the LLM itself and, as said, in case of violation stopping\nthe stream of answer. It is called \u003cem\u003esecond toughts\u003c/em\u003e because the effect seems that the LLM\nhas changed its mind and answer.\u003c/p\u003e","title":"AI Security: Second Thoughts"},{"content":"Introduction The template pattern aims to use the LLM to implement a formatted output starting from a precise request and an associated template with placeholder. The LLM will then iterate the template fulfilling the answer. The template pattern can be useful to generate fake or real data, provide options and lists. A key for executing the prompt is clarifying where the patterns are. So the two main statement that the LLM has to understand are:\nThe LLM is receving a template The LLM has to fill the placeholders What is the format of the placeholders The Prompt To use the pattern a contextual statemen should be made before any instruction. In literature this contextual statement is defined in the following way\nI am going to provide a template for your output [PLACEHOLDERS] is my placeholder for content Try to fit the output into one or more of the placeholders that I list Please preserve the formatting and overall template that I provide. This is the template [Format of the template] (OPTIONAL) According to each request there should be one or many placeholders.\nHow to use the prompt The prompt requires some preparation, mostly in the definition of the template with the placeholder.\nExample A possible example can be the following\nCreate a list of 5 objects that I need to use to deep cleaning my apartment. I\u0026#39;m going to provide a template for your output. [PLACEHOLDER] are my placeholders for each element. Try to fit the output into one or more of the placeholders that I list. Please preserve the formatting and overall template that I provide. This is the template: Room [ROOM THAT I NEED TO LIST], Item [ITEM THAT I NEED],quantity [QUANTITY THAT I NEED] Please format in a table mode. Some Tests Let\u0026rsquo;s run some tests across all the LLM and see if the template is maintained.\nChatGPT 5.2 ChatGPT with the prompt above:\nRoom [ROOM THAT I NEED TO LIST]\tItem [ITEM THAT I NEED]\tquantity [QUANTITY THAT I NEED] Kitchen\tDegreaser Spray\t2 Bathroom\tLimescale Remover\t1 Living Room\tMicrofiber Cloths\t6 Bedroom\tVacuum Cleaner Bags\t3 All Rooms\tDisinfectant Wipes\t4 The pattern is applied and, interesting point of view, ChatGPT is reporting back the placeholders.\nClaude - Haiku 4.5 Claude Haiku 4.5 - Same input prompt\n| Room | Item | Quantity | |------|------|----------| | Kitchen | Degreaser spray. | 1 bottle | | Bathroom | Toilet brush and plunger | 1 each | | Living Room | Microfiber cloths | 10-15 pieces | | Bedrooms | Vacuum bags or filters | 2-3 sets | | All Rooms | All-purpose cleaner | 2 bottles | Also here the pattern is fulfilled.\nMistral Mistral AI, same prompt\nDeep Cleaning Essentials | Room | Item | Quantity | |--------------|-------------------------------|----------| | Whole House | Microfiber cloths | 5-10 | | Kitchen | Degreaser cleaner | 1 bottle | | Bathroom | Disinfectant spray/wipes | 1 bottle | | Floors | Vacuum cleaner (with HEPA) | 1 | | Whole House | All-purpose cleaner | 1 bottle | Conclusions The Template Pattern is probably one of the simplest and most effective pattern. Spending a bit of time preparing the prompt with template, placeholders and format can then easily generate structured and usable output.\n","permalink":"https://www.fanfani.net/posts/07-february-2026/","summary":"\u003ch3 id=\"introduction\"\u003eIntroduction\u003c/h3\u003e\n\u003cp\u003eThe template pattern aims to use the LLM to implement a formatted output starting from  a precise\nrequest and an associated template with placeholder. The LLM will then \u003cem\u003eiterate\u003c/em\u003e the template fulfilling the answer.\nThe template pattern can be useful to generate fake or real data, provide options and lists.\nA key for executing the prompt is clarifying where the patterns are. So the two  main statement that the LLM\nhas to understand are:\u003c/p\u003e","title":"Prompt Engineering: Template Pattern"},{"content":"Introduction The Outline Expansion Pattern aims to use the LLM to provide a hierarchical list of possibilities and deep dive in one or more point of the list, creating a narrowed set of possibilities. The pattern relies on the concept of outline expansion, essentially the creation of:\nAn initial list of major topics and arguments For each element of the list a sublist of topics related to the main one Until the granularity of the information needed is reached. The outline expansion is considered a writing technique.\nThe Prompt The generic structure of the prompt is the following:\nAct as an outline expander. Generate a bullet point outline based on the input that I give you and then ask me for which bullet point you should expand on. Create a new outline for the bullet point that I select. At the end, ask me for what bullet point to expand next. - (OPTIONAL) If I tell to \u0026#34;go back\u0026#34; show the previous outline. - (OPTIONAL) The bullet point list can have at most [MAX AMOUNT] of bullets. - (OPTIONAL) Each bullet can have at most [MAX AMOUNT] sub bullets. - (OPTIONAL) The bullets should be numbered using the pattern [A-Z].[i-v].[* through ****] Ask me for what to outline. The main caveats are related to:\nThe number of elements in a list can easily become overwhelming The number of levels in the outline can become not-manageable. With these two points in mind the pattern can help to create an initial skeleton of ideas, refine them and iterate. It is quite useful introducting the possibility to go back in the navigation, asking to the LLM to re-show the previous level of the outline.\nTo be clear the patter will only provide the navigation, not ending the list on a specific point neither expanding the request with extra description.\nHow to use the Prompt The prompt must be setup as contextual prompt before any session. After that the first request is made and then, via choice of which bullet-point should be expanded, the user can navigate the structure.\nSome Tests Let\u0026rsquo;s try the prompt in all its shining with all the optional fields.\nThe tests will use\nAct as an outline expander. Generate a bullet point outline based on the input that I give you and then ask me for which bullet point you should expand on. Create a new outline for the bullet point that I select. At the end, ask me for what bullet point to expand next. If I tell to \u0026ldquo;go back\u0026rdquo; show the previous outline. The bullet point list can have at most 4 of bullets. Each bullet can have at most 3 sub bullets. 3 The bullets should be numbered using the pattern [A-Z].[i-v].[* through ****] Ask me for what to outline. and suggest me a menu for dinner for few friends at my apartment\nas a request. The idea behind the test is checking in how many steps the LLM will provide a list that can be useful or instead will enter in an endless loop.\nGoogle Gemini Putting the prompt in Gemini\nThe answer is the expected one. With the associated question:\nThe LLM provides a set of possibilities, and selected one, provides the expansion in sub bullets. Putting aside the quality of the food suggested the navigation landed in a brief set of instruction on how to cook\nAlso back works fine.\nClaude - Haiku 4.5 Haiku 4.5 works in similar way:\nPrompt and request\noutline and expansion, including back\nNavigation and results\nMistralAI Mistral AI provides the most compact and useful navigation, landing on the instruction on how to cook in a easy and quick way. Also here the \u0026ldquo;back\u0026rdquo; function works.\nConclusions I\u0026rsquo;m surprised about this pattern, as I was thinking that was just landing in an endless list of bullet points. Assuming a clear question the pattern can be really useful to define, refine and reach te information needed.\nAs rule of thumb 3 to 4 bullet points are manageable and, if not landing on a good answer in two or three steps probably something is going wrong.\n","permalink":"https://www.fanfani.net/posts/02-february-2026/","summary":"\u003ch3 id=\"introduction\"\u003eIntroduction\u003c/h3\u003e\n\u003cp\u003eThe \u003cem\u003eOutline Expansion Pattern\u003c/em\u003e aims to use the LLM to provide a hierarchical list of possibilities\nand deep dive in one or more point of the list, creating a narrowed set of possibilities.\nThe pattern relies on the concept of \u003cem\u003eoutline expansion\u003c/em\u003e, essentially the creation of:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eAn initial list of major topics and arguments\u003c/li\u003e\n\u003cli\u003eFor each element of the list a sublist of topics related to the main one\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eUntil the granularity of the information needed is reached.\nThe outline expansion is considered a writing technique.\u003c/p\u003e","title":"Prompt Engineering: Outline Expansion Pattern "},{"content":"Introduction The Alternative Approach Pattern aims to use the LLM to define alternative possibilies either from the prompt itself or from the answer of the LLM. As a more general view the patterns aims to help the users to define the underlying problems that they are trying to solve.\nThe Prompt The generic structure of the prompt is the following:\nIf there are alternative ways to accomplish [THE INSTRUCTION] that I give you, list the best alternate approaches - (OPTIONAL) include the original way that I asked - (OPTIONAL) compare/contrast the pros and cons of each approach - (OPTIONAL) prompt me for which approach I would like to use the prompt it is essentially telling to the LLM to provide new possibilities.\nTwo commom implementations are available in the literature:\nVersion One\nFrom now on, whenever I ask something, follow this approach: - Put my original question exactly as I phrased it. - Present an alternative approach or perspective that could also effectively address the question. - Provide a clear and concise comparison of the pros and cons of each approach. - Ask me which approach I would like you to use to respond to my original question. Version Two\nFor anything that I ask you to write, determine the underlying problem that I am trying to solve and how I am trying to solve it. List at least one alternative approach to solve the problem and compare the approach with the original approach implied by my request to you. These are the two main approches suggested.\nHow to use the Prompt There is not a real rule on which one of the two approaches is better. While the first one is more focused on the prompt itself, the second aims to push the LLM to provide different options in the answer. Both need to be set as contextual prompt before any session.\nSome Tests Let\u0026rsquo;s play with the pattern, in both modes, using quite generic question and see what are the answers. The idea is comparing the answer and see how the LLMs \u0026ldquo;react\u0026rdquo; to the instructions. The question will be\nPlan me a week of holiday in Madeira\nChatGPT 5.2 Let\u0026rsquo;s validate on chatGPT the approach, both focused on the prompt and the answer.\nAlternative Approach - Version 1 Given the version 1 of the prompt above ChatGPT provides a quite standard and expected acknoledgement\nGot it — I’ll follow that approach from now on. 👍 I’ve saved this as your preferred response format for future questions. Whenever you ask something next, I’ll\nQuote your original question exactly as you wrote it Offer an alternative approach or perspective Compare the pros and cons of each Ask which approach you want me to use before proceeding Whenever you’re ready, ask your next question and we’ll try it out. Then with the generic question above:\nYour original question (exactly as phrased): “Plan me a week of holiday in Madeira” Two possible ways we can approach this\nApproach A — Ready-made itinerary I create a day-by-day plan for 7 days in Madeira with specific activities, sights, towns, and pacing.\nPros\nVery concrete and easy to follow Saves you time and decision fatigue Great if you want a “just tell me what to do” plan Cons\nLess flexible if your interests differ May include things you’d personally skip Needs assumptions (budget, activity level, travel style) Approach B — Modular \u0026amp; customizable plan I outline themes and options for each day (nature, food, relaxation, adventure), plus recommendations, and you assemble the week based on what excites you most.\nPros\nHighly flexible and personal Adapts easily to weather, energy, or companions Better if you like choice and spontaneity Cons\nRequires a bit more decision-making Less “plug-and-play” than a fixed itinerary Might feel abstract if you want firm plans So the LLM is providing two possible approaches based on the contextual prompt and the specific request.\nAlternative Approach - Version 2 With the second version of the prompt and the same question:\nYou asked: “Plan me a week of holiday in Madeira.” I’ll handle this in the structure you asked for.\nUnderlying problem you’re trying to solve You want a clear, low-friction plan for a 7-day trip to Madeira that helps you: Use your limited time well See the island’s highlights without overthinking logistics Balance nature, relaxation, and culture In short: “Tell me what to do each day so I can enjoy Madeira without planning paralysis.” The approach you’re implicitly taking By asking me to plan a week, you’re implying a day-by-day itinerary approach: Fixed structure (Day 1, Day 2, etc.) Pre-selected sights and activities Optimized for coverage and flow Strengths of this approach\nEasy to follow Reduces decision fatigue Great for first-time visitors Alternative approach Theme-based + flexible days approach. Instead of rigid daily plans: Group activities by theme (Nature, Sea, Culture, Chill) Assign each day a focus rather than a checklist Decide details the night before based on weather and mood The main different is in the implicit assumption that the LLM has made on my answer and used to determine the following possible approaches.\nClaude - Haiku 4.5 Let\u0026rsquo;s see how claude Haiku 4.5 will answer on both approaches:\nAlternative Approach - Version 1 Same prompt version and same question, let\u0026rsquo;s see the answer\nI understand. From now on, when you ask me something, I\u0026rsquo;ll:\nRepeat your original question exactly as you phrased it Suggest an alternative approach that could also address your question Compare the pros and cons of both approaches clearly Ask which approach you\u0026rsquo;d prefer me to use I\u0026rsquo;m ready to go! Feel free to ask your first question whenever you\u0026rsquo;re ready. Your original question: \u0026ldquo;Plan me a week of holiday in Madeira\u0026rdquo;\nApproach 1: Comprehensive Itinerary Create a detailed day-by-day itinerary with specific activities, restaurants, accommodations, and logistics (times, costs, booking links, transport details). This would be structured and ready to execute.\nApproach 2: Framework \u0026amp; Recommendations Provide a flexible framework organized by interest areas (nature/hiking, culture, beaches, food, nightlife) with curated suggestions you can mix and match based on your preferences, budget, and pace—giving you more control to customize your own itinerary.\nComparison:\nAspect Approach 1 Approach 2 Pros Ready-to-follow; minimal planning needed; good pacing Personalized; adaptable to your interests; lets you discover; more flexible Cons Less flexible; may not match your preferences; feels prescriptive Requires more effort to organize; need to make decisions Best for People who want it done for them People who like customizing their trips Which approach would you prefer? Or would you like me to combine elements of both?\nThe Claude answer is applying the correct pattern with a shorter comparison on a table form.\nAlternative Approach - Version 2 Again, same prompt version and same question, let\u0026rsquo;s see the answer:\nI appreciate that instruction—it\u0026rsquo;s a thoughtful way to approach problem-solving. I\u0026rsquo;ll do my best to follow it for writing requests going forward. To clarify how I\u0026rsquo;ll apply it: When you ask me to write something, I\u0026rsquo;ll:\nIdentify the underlying problem you\u0026rsquo;re trying to solve and the approach you\u0026rsquo;re implying Suggest at least one alternative approach that could address that problem differently Compare the approaches — highlighting tradeoffs, pros/cons, and when each might be most appropriate I\u0026rsquo;ll present these alternatives naturally within my response rather than in a rigid list format, unless you prefer otherwise. Feel free to let me know if you\u0026rsquo;d like me to adjust how I do this, or if there are specific writing tasks you\u0026rsquo;d like me to tackle with this approach in mind!\nAnd, after the question\n\u0026ldquo;Plan me a week of holiday in Madeira\u0026rdquo;\nHere the answer\nI\u0026rsquo;d like to understand what you\u0026rsquo;re really looking for before planning this out.\nThe underlying problem seems to be: you want a structured, enjoyable week in Madeira that makes good use of your time and budget. Your implied approach is probably a day-by-day itinerary covering major attractions and activities. But let me suggest an alternative framing: Instead of a fixed itinerary, what if we first clarified what kind of holiday you actually want?\nHere are some different approaches:\nStructured itinerary approach (what you likely asked for): A day-by-day plan hitting popular spots like Funchal, Pico Ruivo hikes, coastal villages, etc. This works well \u0026gt;if you like knowing exactly what\u0026rsquo;s happening, enjoy checking off experiences, and want to see the \u0026ldquo;must-sees.\u0026rdquo; Flexible framework approach: Instead, I could suggest 4-5 core experiences (hiking, local food, beach/coastal time, cultural exploration, relaxation) and let you mix them \u0026gt;based on weather and mood. This works better if you prefer spontaneity and want to adapt to how you\u0026rsquo;re feeling. Interest-driven approach: If I knew whether you\u0026rsquo;re primarily into hiking, beaches, culture, wine, food, relaxation, or photography, I could tailor recommendations much \u0026gt;more precisely rather than giving a generic overview. Before I plan it, could you tell me:*\nWhat\u0026rsquo;s your main draw to Madeira? (nature/hiking, beaches, culture, food, or a mix?) Are you traveling solo, with a partner, or with family? Do you prefer a structured schedule or flexibility? Any physical limitations on activities like hiking? This will help me create something actually tailored to your holiday rather than a generic week. An interesting point Claude is asking for extra info to fine tune the possible options.\nMistral Alternative Approach - Version 1 Same prompt same question:\nUnderstood! From now on, I’ll structure my responses to your questions as follows:\nYour original question* (quoted exactly as you phrased it) An alternative approach or perspective* (how else the question could be addressed) Comparison of pros and cons (for both your original approach and the alternative) Your choice (I’ll ask which approach you’d like me to use for the response) This way, you can decide how you’d like me to proceed. Go ahead and ask your question! Your Original Question: \u0026ldquo;Plan me a week of holiday in Madeira.\u0026rdquo;\nAlternative Approach: Instead of a traditional itinerary focused on sightseeing and activities, I could design your week around themes (e.g., nature immersion, cultural deep dive, relaxation, or adventure). This would allow you to experience Madeira in a more personalized way, depending on your interests.\nMistral is pretty short and dry in the answer, providing the two possible approaches and leaving to the user the choice.\nAlternative Approach - Version 2 Last test. Second version of prompt and same question on Mistral.\nUnderlying Problem \u0026amp; Your Approach Problem: You want to plan a relaxing and enjoyable week-long holiday in Madeira, likely seeking a mix of nature, culture, local experiences, and relaxation. Your Approach: You’re asking for a structured itinerary to maximize your time and ensure you don’t miss out on key attractions. Alternative Approach Alternative: Instead of a fixed day-by-day plan, you could opt for a flexible, theme-based itinerary (e.g., \u0026ldquo;Adventure Day,\u0026rdquo; \u0026ldquo;Cultural Day,\u0026rdquo; \u0026ldquo;Relaxation Day\u0026rdquo;). This allows you to adapt based on weather, energy levels, or spontaneous discoveries. Comparison:\nFixed Itinerary Theme-Based Itinerary ✅ Ensures you see all major attractions. ✅ More adaptable to mood/weather. ✅ Efficient use of time. ✅ Less pressure to stick to a schedule. ❌ Can feel rushed or rigid. ❌ Risk of missing out on must-see spots. [\u0026hellip;]\nWith the usual table at the bottom.\nConclusions The prompt works, mostly in the first approach while the second seems less effective. All the LLM make try to analyse the underlying problem in the second version. Interesting is that Claude is asking, in the second version, further info to fine tune the answer.\n","permalink":"https://www.fanfani.net/posts/01-february-2026/","summary":"\u003ch3 id=\"introduction\"\u003eIntroduction\u003c/h3\u003e\n\u003cp\u003eThe \u003cem\u003eAlternative Approach  Pattern\u003c/em\u003e aims to use the LLM to define alternative possibilies either from the prompt itself or from the answer of the LLM.\nAs a more general view the patterns aims to help the users to define the underlying problems that they are trying to solve.\u003c/p\u003e\n\u003ch3 id=\"the-prompt\"\u003eThe Prompt\u003c/h3\u003e\n\u003cp\u003eThe generic structure of the prompt is the following:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;\"\u003e\u003ccode class=\"language-markdown\" data-lang=\"markdown\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003eIf there are alternative ways to accomplish [THE INSTRUCTION] that I give you, list the best alternate approaches\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#66d9ef\"\u003e-\u003c/span\u003e (OPTIONAL) include the original way that I asked\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#66d9ef\"\u003e-\u003c/span\u003e (OPTIONAL) compare/contrast the pros and cons of each approach\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#66d9ef\"\u003e-\u003c/span\u003e (OPTIONAL) prompt me for which approach I would like to use\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003ethe prompt it is essentially telling to the LLM to provide new possibilities.\u003c/p\u003e","title":"Prompt Engineering: Alternative Approach Pattern"},{"content":"Introduction The Semantic Filter Pattern aims to use the LLM to filter, from a generic text, a subset of information. The filter can be applied either to \u0026ldquo;filter in\u0026rdquo; so keeping, or \u0026ldquo;filtering out\u0026rdquo;, so removing the specific content. The idea is that:\nThe LLM will receive a filter rule The LLM will receive a text The outcome of the LLM will be either the filtered text or the \u0026ldquo;remaining\u0026rdquo; text if the request is removing the filtered content. The Prompt The prompt is based on a contextual statement given before any input that can be summarised in\nFilter this information to remove [RULE] when the intention is removing the filtered content\nOr, alternately\nFilter this information to keep only [RULE] when the intention is keeping the filtered content\nor, alternately\nFilter this information to [DO SOME ACTION] based on [RULE] if the request is altering the data.\nSome examples The pattern can be used to keep or to remove a set of personal data. As example, in case of removal or alteration\nClean and anonymize the following data by removing all personally identifiable information and other details that could lead to identification. Remove any sentitive information linked with healtcare or private data Remove all the addresses and the location of the customers\nor, in case of keeping\nKeep only the critical feedback from the customers Keep only the jokes from this text Keep only the quotes of famous people from this text\nand so on. What is important is that the rule is specificically defined for the domain associated on the main text.\nAs example, in healthcare, the focus will be on diseases, personal data and private NIN numbers, while in a sales document focus of the filter will be on customers\u0026rsquo;s data and revenues.\nHow to use the prompt First create the filter rule, either targeting to keep or remove some information but absolutely tailored to the domain of the data that needs to be filtered.\nThe give the instruction to the LLM to apply the filter and compare the results\nSome Tests To run some tests we need some data. As example the following (fake) list of patients, with bio and disease will provide the basic information.\nMaria Thompson (Age 45) – Type 2 Diabetes Maria works as a school administrator and enjoys gardening and cooking for her family. She lives in a suburban neighborhood and is actively trying to improve her lifestyle through daily walks and healthier eating habits. James Carter (Age 62) – Hypertension James is a retired factory supervisor who now spends his time fishing and woodworking. He lives with his wife and enjoys quiet mornings reading the newspaper and tending to his backyard plants. Aisha Khan (Age 29) – Asthma Aisha is a graphic designer who works remotely and loves digital art and photography. She lives in an apartment in the city and enjoys yoga, journaling, and café-hopping on weekends. Robert Miller (Age 54) – Chronic Obstructive Pulmonary Disease (COPD) Robert is a former construction worker who now runs a small home repair business. He enjoys watching classic movies, fixing old radios, and spending time with his grandchildren. Elena Garcia (Age 37) – Rheumatoid Arthritis Elena is a community librarian who loves organizing reading programs for children. She enjoys painting, nature walks, and volunteering at local cultural events. David Wilson (Age 71) – Alzheimer’s Disease David is a retired history teacher who once traveled extensively across Europe. He now lives with his daughter and enjoys listening to classical music and looking through old photo albums. Nina Patel (Age 24) – Iron Deficiency Anemia Nina is a university student studying biomedical engineering. She enjoys volunteering, participating in student clubs, and exploring new cafés around campus. Michael Brown (Age 48) – Coronary Artery Disease Michael runs a small logistics company and spends most of his free time coaching his son’s soccer team. He enjoys road trips and restoring old cars in his garage. Sophia Lee (Age 33) – Hypothyroidism Sophia is a freelance translator who works with multiple international clients. She enjoys language learning, journaling, and practicing calligraphy in her free time. Omar Hassan (Age 58) – Chronic Kidney Disease (Stage 3) Omar owns a neighborhood grocery store and is well known in his community. He enjoys early morning walks, reading historical novels, and spending evenings with his family. While the filtering rule will be:\nAnonymize the data by removing all personally identifiable information and other details that could lead to identification of the subjects\nKeep only the critical diseases from the following data\nBoth rules will be given to the LLM before receiving the list above.\nChatGPT 5.2 giving the list and the first rule to chatGPT the result is the following\nHere’s an anonymized version of your data. I’ve removed names, specific occupations, exact locations, and other identifying personal details \u0026gt;while keeping the relevant medical information and general lifestyle context:\nSubject 1 (Age 45) – Type 2 Diabetes The individual works in an administrative role and enjoys gardening and cooking. They live in a suburban area and are actively improving their lifestyle through daily walks and healthier eating habits. Subject 2 (Age 62) – Hypertension The individual is retired and spends time on hobbies like fishing and woodworking. They live with family and enjoy quiet mornings reading and tending to their backyard. Subject 3 (Age 29) – Asthma The individual works remotely in a creative field and enjoys digital art and photography. They live in an urban apartment and like yoga, journaling, and visiting cafés on weekends. Subject 4 (Age 54) – Chronic Obstructive Pulmonary Disease (COPD) The individual previously worked in construction and now runs a small home-based business. They enjoy watching classic movies, fixing electronics, and spending time with grandchildren. Subject 5 (Age 37) – Rheumatoid Arthritis The individual works in a community-focused role and enjoys organizing programs for children. They also like painting, nature walks, and volunteering at local cultural events. Subject 6 (Age 71) – Alzheimer’s Disease The individual is retired from teaching and previously traveled extensively. They live with family and enjoy listening to classical music and looking through old photo albums. Subject 7 (Age 24) – Iron Deficiency Anemia The individual is a university student studying biomedical engineering. They enjoy volunteering, participating in student clubs, and exploring new cafés around campus. Subject 8 (Age 48) – Coronary Artery Disease The individual runs a small business and spends free time coaching youth sports. They enjoy road trips and restoring old vehicles. Subject 9 (Age 33) – Hypothyroidism The individual works as a freelance translator. They enjoy language learning, journaling, and practicing calligraphy. Subject 10 (Age 58) – Chronic Kidney Disease (Stage 3) The individual owns a local store and is active in the community. They enjoy morning walks, reading, and spending time with family. If you want, I can also generalize the ages into ranges and remove even occupations to make it fully de-identified for research purposes. \u0026gt;This would make it extremely difficult to trace back to anyone. So the outcome is a quite anonymous list. Interesting the note that is asking if we want to push further the semantic filtering.\nOn the second test, the one linked with keeping the content, with the same list, the answer is\nMaria Thompson\t45\tType 2 Diabetes\tSchool administrator\tGardening, cooking, daily walks, healthier eating James Carter\t62\tHypertension\tRetired factory supervisor\tFishing, woodworking, reading, gardening Robert Miller\t54\tCOPD\tHome repair business\tClassic movies, fixing radios, grandchildren David Wilson\t71\tAlzheimer’s Disease\tRetired history teacher\tClassical music, photo albums, family Michael Brown\t48\tCoronary Artery Disease\tLogistics company owner\tCoaching soccer, road trips, restoring cars Omar Hassan\t58\tChronic Kidney Disease (Stage 3)\tGrocery store owner\tMorning walks, historical novels, family time So a list of patiens with, according to the LLM, critical diseases.\nClaude - Haiku 4.5 Claude Haiku 4.5, with the same list and the first filter is answering:\nSubject P001 – Age Group: 40s – Type 2 Diabetes Administrative role. Engages in outdoor activities and cooking. Attempting lifestyle modifications through regular exercise and dietary changes. Subject P002 – Age Group: 60s – Hypertension Retired from supervisory position. Leisure activities include fishing and woodworking. Living with spouse. Enjoys reading and gardening. Subject P003 – Age Group: 20s – Asthma Design professional working remotely. Creative hobbies include digital art and photography. Urban dwelling. Physical activities include yoga and journaling. Subject P004 – Age Group: 50s – Chronic Obstructive Pulmonary Disease (COPD) Former construction worker. Currently self-employed in repair services. Hobbies include classic films and electronics restoration. Has grandchildren. Subject P005 – Age Group: 30s – Rheumatoid Arthritis Works in library services. Organizes community programs. Engages in artistic pursuits, outdoor activities, and volunteer work. Subject P006 – Age Group: 70s – Alzheimer\u0026rsquo;s Disease Retired educator. Previously traveled internationally. Currently lives with adult family member. Enjoys classical music and reminiscence activities. Subject P007 – Age Group: 20s – Iron Deficiency Anemia Student in biomedical field. Participates in volunteer work and student organizations. Enjoys social activities and exploration of local venues. Subject P008 – Age Group: 40s – Coronary Artery Disease Self-employed in transportation/logistics sector. Engages in youth coaching activities. Leisure interests include travel and vehicle restoration. Subject P009 – Age Group: 30s – Hypothyroidism Language services professional. Hobbies include language learning, writing, and artistic pursuits. Subject P010 – Age Group: 50s – Chronic Kidney Disease (Stage 3) Business owner in retail/grocery sector. Morning exercise routine. Enjoys reading and family time.Haiku It is interesting that Claude returns also the specific fields that haven been semantically filtered\nFiltered\nAll names (replaced with Subject IDs: P001-P010) Specific ages (converted to age groups like \u0026ldquo;40s,\u0026rdquo; \u0026ldquo;60s\u0026rdquo;) Specific job titles and company details (generalized to role types) Geographic identifiers (suburban neighborhood, city, campus, community references) Family relationships and specific details (daughter, wife, grandchildren) Hobbies that could narrow identification (e.g., \u0026ldquo;school administrator\u0026rdquo; → \u0026ldquo;Administrative role\u0026rdquo;) Location-specific activities (reading newspaper, café names) Preserved:\nMedical diagnoses (essential for healthcare purposes) General lifestyle and activity levels Functional information relevant to health management Age ranges (useful for epidemiological analysis) Broad occupational categories On the second test the answer is the following\nBased on medical severity and urgency, here are the critical diseases from the dataset:\nDavid Wilson (Age 71) – Alzheimer\u0026rsquo;s Disease - Neurodegenerative disease with progressive cognitive decline and high mortality risk Robert Miller (Age 54) – Chronic Obstructive Pulmonary Disease (COPD) - Serious respiratory condition with significant mortality risk and quality of life impact Michael Brown (Age 48) – Coronary Artery Disease - Life-threatening cardiovascular condition with high risk of heart attack and sudden death Omar Hassan (Age 58) – Chronic Kidney Disease (Stage 3) - Progressive condition that can lead to kidney failure and require dialysis or transplantation Interesting thing, in this case, the result is slighty different from ChatGPT.\nMistral Same data and same questions to Mistral, and the following table for the first filter:\nAnonymized Subject Data\nSubject ID Age Condition Occupation Type Hobbies/Interests (Generalized) Living Situation (Generalized) 001 45 Type 2 Diabetes Administrative Outdoor activities, cooking Suburban area 002 62 Hypertension Retired (supervisory) Crafts, reading, gardening With spouse 003 29 Asthma Creative/Remote Work Art, wellness activities, social outings Urban apartment 004 54 Chronic Obstructive Pulmonary Disease (COPD) Skilled Trade/Small Business Classic entertainment, hands-on hobbies, family time Home-based 005 37 Rheumatoid Arthritis Public Service Creative arts, outdoor activities, community involvement Urban area 006 71 Alzheimer’s Disease Retired (Education) Music, reminiscing With family 007 24 Iron Deficiency Anemia Student Volunteering, social activities, exploring local venues Campus area 008 48 Coronary Artery Disease Business Owner Sports coaching, automotive hobbies, travel Suburban area 009 33 Hypothyroidism Freelance/Linguistic Language arts, writing, creative hobbies Urban area 010 58 Chronic Kidney Disease (Stage 3) Retail/Small Business Walking, reading, family time Community-focused Provides a table with the filtered and anonymized data.\nOn the second filter\nCritical Diseases\nName Age Critical Disease Robert Miller 54 Chronic Obstructive Pulmonary Disease (COPD) David Wilson 71 Alzheimer’s Disease Michael Brown 48 Coronary Artery Disease Omar Hassan 58 Chronic Kidney Disease (Stage 3) the answer of Mistral regarding the critical diseases is the same of Claude.\nConclusions The semantic filter pattern aims to identify, keep, alter or remove some information from data according to a well define rule in the filter itself. While the removal of information seems to be a easier filter, the keeping must be really well-defined. In any case the result must be checked as the LLM could not get the exact semantic meaning suggested.\n","permalink":"https://www.fanfani.net/posts/21-january-2026/","summary":"\u003ch3 id=\"introduction\"\u003eIntroduction\u003c/h3\u003e\n\u003cp\u003eThe \u003cem\u003eSemantic Filter Pattern\u003c/em\u003e aims to use the LLM to filter, from a generic text, a subset of information.  The filter can be applied either to \u0026ldquo;filter in\u0026rdquo; so keeping, or \u0026ldquo;filtering out\u0026rdquo;, so removing\nthe specific content.  The idea is that:\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003eThe LLM will receive a filter rule\u003c/li\u003e\n\u003cli\u003eThe LLM will receive a text\u003c/li\u003e\n\u003cli\u003eThe outcome of the LLM will be either the filtered text or the \u0026ldquo;remaining\u0026rdquo; text if the request is removing the filtered content.\u003c/li\u003e\n\u003c/ol\u003e\n\u003ch3 id=\"the-prompt\"\u003eThe Prompt\u003c/h3\u003e\n\u003cp\u003eThe prompt is based on a \u003cem\u003econtextual statement\u003c/em\u003e given before any input that can be summarised in\u003c/p\u003e","title":"Prompt Engineering: Semantic Filter Pattern"},{"content":"Introduction I was lucky to be involved in a workshop on how to use Kiro and playing with it. As a non developer I have to say that the results are impressive. To quote I had the feeling that Any sufficiently advanced technology is indistinguishable from magic\nWhat is Amazon Kiro? What is Amazon Kiro? Amazon Kiro is one of the new cool tools focused on specs-driven development where, according to this good very post the main focus is not on the code itself, but it is on the generation of a set of artifacts, commonly markdown documents, associated with a dedicated workflow.\nThe sum of the documents plus the workflow in the tool then creates the code itself.\nKiro, like other tools, tried to\nStructure the vibe coding approach with a set of artifacts Track the modifications Improve the interaction between the human and the tools. Under the hood the usual structure of agents, LLM and so on. But it is not that the point. The main point is trying to move left (or upstream) the interactions and the inputs.\nKiro\u0026rsquo;s workflow is based on three major steps:\nRequirements A document contains essentially a set of user\u0026rsquo;s stories Design A document containins the definition of the software, some building blocks and tech specs Tasks All the actions, mandatory or optionals, that need to be done to complete the code Now the question is: Who is writing these documents? The answer is pretty simple, it is Kiro itself, having the human just reading, or trying to read, the documents and then click and confirm.\nThe Project - A Python reader for GGUF With few Kiro\u0026rsquo;s credits and a rainy afternoon I decided to try to generate a python script able to read and explore the GGUF file format. For context the GGUF (GTP-Generated Unified Format) is the file format developed by the llama.cpp team and used in the LLM community an for store and save data of an LLM. To simplify a lot it is a sort of massive zip file containing metadata and the weights of the llm.\nFun fact: the magic numbers of the file are the letters GGUF themselves.\nThere are readers in the community on the GGUF file but building one was not the scope, it was just playing with Kiro.\nWhat Kiro Asked Fresh of about 500 credits of Kiro\u0026rsquo;s code I just wrote:\nare you familiar with the GGUF file format, the one used for saving LLM?\nGetting back\nYes, I\u0026rsquo;m familiar with GGUF (GPT-Generated Unified Format). It\u0026rsquo;s a binary format designed for storing and distributing large language models, created by Georgi Gerganov as part of the llama.cpp project. Key things about GGUF:\nSuccessor to GGML: It replaced the older GGML format, addressing its limitations with better extensibility and metadata support Efficient storage: Uses quantization techniques (like Q4_K_M, Q5_K_S, Q8_0) to compress models while maintaining reasonable quality, making large models more accessible Self-contained: Stores model weights, architecture details, tokenizer info, and metadata all in a single file Cross-platform: Designed to work across different systems and hardware, particularly optimized for CPU inference through llama.cpp Metadata-rich: Includes comprehensive information about the model (architecture, parameters, quantization method, etc.) in a structured header The format is widely used in the open-source LLM community, especially for running models locally with tools like llama.cpp, Ollama, LM Studio, and similar inference engines. Are you looking to work with GGUF files in some way? I can help with parsing, converting, or building tools around this format. And then I had this conversation:\n[me] ok can you please create a spec for a python script able to read all the data from a GGUF file?\n[Kiro] I see you\u0026rsquo;re working on a task that\u0026rsquo;s optimized for spec sessions. Would you like to start a dedicated spec session for the best experience?\n[me] Yes\n[Kiro] I\u0026rsquo;ll create a spec for a Python script that can read GGUF files. Let me start by gathering requirements from you and then we\u0026rsquo;ll build out a comprehensive design. I\u0026rsquo;ll help you create a spec for a Python script that can read all data from GGUF files. Let me start by gathering some information about the GGUF format to ensure we have accurate requirements.\nand I said yes.\nWhat Kiro did - The Specifications Then Kiro started and, quite magically, found a bunch of urls with specs and documentation.\nAfter that, in less than a minute, Kiro came back with the first document the requirements document , asking to approve. I did.\nAnother round, another cycle of Kiro and then the design document came back, again, asking for approval. I approved again\nLast run, Kiro generated the task document with 11 tasks, each one with many subtasks. Kiro asked me to Mark all tasks required and I did\nWhat Kiro did - The Coding After that I realised that I let slip the dogs of wars. For a while an endless set of approve actions like running python scripts, shell scripts and validation. Kiro ran for about 14 hours but hanged, so the net time was, as far as I can see, about 2 only hours. Most of the work was writing tests on the script itself, crunching about 133 Kiro credits.\nWhat Kiro did - The Outcome The outcome of the coding is impressive\n[The reader] (https://github.com/andreaf/gguf-reader/blob/master/gguf_reader.py ) a full python library containg all the code [The tests] (https://github.com/andreaf/gguf-reader/blob/master/test_gguf_reader.py ) more than 600 lines of tests Plus artifacts, Readme and examples. All fresh and all well formatted.\nThe Results The result is GGUF-Reader my first and last project on github, including documentations, examples, howto and all the tests. I haven\u0026rsquo;t run it as, every creator, I\u0026rsquo;m scared that my creature will turn in some monster or simply having bugs. The code will stay there, sleeping and dreaming electric sheep.\nConclusions Coming back to the initial quote Any sufficiently advanced technology is indistinguishable from magic it was fun and magic using Kiro to build an entire project, starting from a couple of lines of english. Will I use it in my day-by-day work? As I\u0026rsquo;m not a coder I don\u0026rsquo;t think so but, maybe, I could have found a new hobby.\n","permalink":"https://www.fanfani.net/posts/20-january-2026/","summary":"\u003ch3 id=\"introduction\"\u003eIntroduction\u003c/h3\u003e\n\u003cp\u003eI was lucky to be involved in a workshop on how to use Kiro and playing with it. As a \u003cem\u003enon developer\u003c/em\u003e I have to say that the results\nare impressive. To quote I had the feeling that \u003cem\u003eAny sufficiently advanced technology is indistinguishable from magic\u003c/em\u003e\u003c/p\u003e\n\u003ch3 id=\"what-is-amazon-kiro\"\u003eWhat is Amazon Kiro?\u003c/h3\u003e\n\u003cp\u003eWhat is Amazon Kiro? Amazon Kiro is one of the new cool tools focused on \u003cem\u003especs-driven development\u003c/em\u003e where, according \u003ca href=\"https://martinfowler.com/articles/exploring-gen-ai/sdd-3-tools.html\" rel=\"noopener noreferrer\" target=\"_blank\"\u003eto this good very post\u003c/a\u003e\n the main focus is not on the code itself, but it is on the generation of a set of artifacts, commonly markdown documents, associated with a dedicated workflow.\u003c/p\u003e","title":"Look mum, I'm a coder - My minutes of glory with Amazon Kiro"},{"content":"Introduction The Tail Generation Pattern is a small utility pattern that can be used to re-iterate, add or confirm some notes at the end of the answer of the LLM. It is more an utility that a real pattern but, in some circustances can be quite useful.\nThe Prompt The basic structure of the prompt, in literature, is the following statement:\nAt the end of your answer, repeat [STATEMENT] and/or ask me for [QUESTION]. Added at the end of the question. A possible example is pretty simple\nFrom now at the end of your answer, repeat my question. Or, alternately\nFrom now at the end of your answer, ask me if I want to know more How to use the prompt The usage of the prompt is pretty simple. It has to be inserted as first request in a session, before any further ask\nSome Tests Testing this prompt is pretty simple, the main validations are:\nThe LLM acknoledges that it will repeat the statement The LLM is correctly repeating the statement at the end ChatGPT 5.2 ChatGPT gets the request\nFrom now at the end of your answer, repeat my question.\nand answers\nGot it — I’ll repeat your question verbatim at the end of every response from now on. From now at the end of your answer, repeat my question.\nbut it entering in an interesting loop, as it repeating the question (From now at the end of your answer, repeat my question) on the answer of the request itself.\nClaude - Haiku 4.5 Claude is smarter, as does not repeat the question itself on the first interaction\nFrom now at the end of your answer, repeat my question.\nI understand! From now on, I\u0026rsquo;ll repeat your question at the end of my responses. Is there anything I can help you with?\nMistral Mistral too has a more user friendly approach in the answer.\nFrom now at the end of your answer, repeat my question.\nUnderstood! I’ll repeat your question at the end of each of my answers. What would you like to know or discuss today?\nConclusions The Tail Generation Pattern is a simple way to add some extra message in the answer of the LLM, used then to manage the real request. Easy to implement and quite useful in interactive context\n","permalink":"https://www.fanfani.net/posts/19-january-2026/","summary":"\u003ch3 id=\"introduction\"\u003eIntroduction\u003c/h3\u003e\n\u003cp\u003eThe \u003cem\u003eTail Generation Pattern\u003c/em\u003e is a small utility pattern that can be used to re-iterate, add or confirm some notes at the end of\nthe answer of the LLM. It is more an utility that a real pattern but, in some circustances can be quite useful.\u003c/p\u003e\n\u003ch3 id=\"the-prompt\"\u003eThe Prompt\u003c/h3\u003e\n\u003cp\u003eThe basic structure of the prompt, in literature, is the following  statement:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;\"\u003e\u003ccode class=\"language-markdown\" data-lang=\"markdown\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003eAt the end of your answer, repeat [STATEMENT] and/or ask me for [QUESTION].\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eAdded at the end of the question. A possible example is pretty simple\u003c/p\u003e","title":"Prompt Engineering: Tail Generation Pattern"},{"content":"Introduction The Fact Check List Pattern it is a valid way to cross-check the output of an LLM assuming that the user has a limited knowledge of the topic asked. The Fact check List provides a list of facts linked with the main request that can be used, by the user, to cross-check or at least compare the main ouput of the LLM. The prompt is inserted before any request as part of the overall session.\nThe Prompt The basic structure of the prompt, in literature, is the following contextual statement:\nGenerate a set of facts that are contained in the output The set of facts should be inserted at POSITION in the output The set of facts should be the fundamental facts that could undermine the veracity of the output if any of them are incorrect In an application can be inserted in this way:\nWhenever you output text, generate a set of facts that are contained in the output. The set of facts should be inserted at the end of the output. The set of facts should be the fundamental facts that could undermine the veracity of the output if any of them are incorrect. In this way the LLM must, for each answer, provide a sort of fact-checking list.\nIt is implicit that the list of facts added to the answer should:\nchecked as first cross-checked with the answer itself How to use the prompt The prompt can be use to cross-check the output of the LLM but also to enrich the answer itself. If used with the POSITION of the fact at the beginning of the answer the risk is that will become a sort of \u0026ldquo;summary\u0026rdquo; of the answer itself. If put at the end of the answer can be used to cross-check step by step the deep dive of the LLM\nSome Tests As said the prompt must be used as a \u0026ldquo;context\u0026rdquo; before any request. The position of the \u0026ldquo;fact\u0026rdquo; can be either at the end of the answer or at the beginning. After each answer is in charge of the user cross-checking the facts with the answer itself.\nChatGPT 5.2 If we prompt ChatGPT with the contextual prompt above before any question\nWhenever you output text, generate a set of facts that are contained in the output. The set of facts should be inserted at the end \u0026gt; of the output. The set of facts should be the fundamental facts that could undermine the veracity of the output if any of them are \u0026gt; incorrect\nthe answer associated is an acknoledge of the request\nUnderstood. I will include a set of fundamental facts at the end of every response I provide, summarizing the key factual claims \u0026gt;contained in the output—specifically those facts whose incorrectness would undermine the response’s veracity. I’ll label this section clearly (e.g., “Facts contained in this response”) and keep it concise and focused on core claims rather \u0026gt;than opinions, interpretations, or advice. Facts contained in this response\nYou requested that every response I generate include a set of fundamental facts contained in the output. I acknowledged and agreed to follow this instruction for future responses. I stated that the facts section will summarize key factual claims whose incorrectness would undermine the response’s accuracy. After that, a generic question, eg:\ntell me how to cook spaghetti carbonara\nwill contain, at the end, a set of associated facts\nSet of Facts\nTraditional carbonara uses eggs, Pecorino Romano, guanciale, and black pepper. Cream is not used in authentic carbonara. The sauce becomes creamy through emulsification of eggs, cheese, and pasta water. Eggs must be added off the heat to avoid scrambling. Guanciale is the traditional meat used in carbonara. Spaghetti carbonara is served immediately after preparation. Claude - Haiku 4.5 Time to test Haiku 4.5 on the same approach\nWhenever you output text, generate a set of facts that are contained in the output. The set of facts should be inserted at the end \u0026gt;of the output. The set of facts should be the fundamental facts that could undermine the veracity of the output if any of them are \u0026gt;incorrect.\nThe answer is similar only, as usual, bit lengthy\nI understand. Going forward, I\u0026rsquo;ll include a set of fundamental facts at the end of my text outputs. These will be the key factual \u0026gt;claims that, if incorrect, would undermine the accuracy of what I\u0026rsquo;ve said. For example, if I explain how photosynthesis works, I might list facts like:\nPlants use chlorophyll to absorb light energy Photosynthesis converts carbon dioxide and water into glucose and oxygen This process primarily occurs in the chloroplasts of plant cells Or if I answer a question about a historical event, I\u0026rsquo;d list the core facts that support my answer. This is a helpful approach because it makes the foundation of my responses transparent and allows you to quickly verify the key \u0026gt;?\u0026gt;claims. If any of those facts turn out to be wrong, you\u0026rsquo;d know the output needs revision.\nI\u0026rsquo;m ready to apply this to future responses. Just let me know if you\u0026rsquo;d like me to adjust how I present these facts (different format, level of detail, etc.).\nFundamental Facts in This Response:\nI will include a set of facts at the end of text outputs going forward These facts represent the core claims that support the accuracy of the output If any of these facts are incorrect, the output\u0026rsquo;s veracity would be undermined Mistral Let\u0026rsquo;s try Mistral AI on the same approach\nWhenever you output text, generate a set of facts that are contained in the output. The set of facts should be inserted at the end \u0026gt;of the output. The set of facts should be the fundamental facts that could undermine the veracity of the output if any of them are \u0026gt;incorrect.\nBetter answer\nUnderstood! From now on, whenever I provide an output, I’ll include a set of fundamental facts at the end. These facts will \u0026gt;be the key elements that, if incorrect, could undermine the accuracy of the information shared. If you have a specific topic or question in mind, feel free to ask!\nConclusions The Fact Check List Pattern can be considered as a safety belt on the answer of the LLM. As side effect it requires extra effort to cross-check the facts themselves. A wrong fact should be a considered as an alarm that also the answer itself could be wrong.\n","permalink":"https://www.fanfani.net/posts/10-january-2026/","summary":"\u003ch3 id=\"introduction\"\u003eIntroduction\u003c/h3\u003e\n\u003cp\u003eThe \u003cem\u003eFact Check List Pattern\u003c/em\u003e it is a valid way to cross-check the output of an LLM assuming that the user has a limited\nknowledge of the topic asked. The  \u003cem\u003eFact check List\u003c/em\u003e provides a list of \u003cem\u003efacts\u003c/em\u003e linked with the main request that can be used, by the user, to cross-check or at least compare the main ouput of the LLM.  The prompt is inserted \u003cem\u003ebefore\u003c/em\u003e any request as part of the overall session.\u003c/p\u003e","title":"Prompt Engineering: Fact Check List Pattern"},{"content":"Introduction The Ask for Input pattern is a nifty little prompt trick to prepare the conversation with an LLM after a more complex pattern. It can be considered just a way to complete the set of initial instruction and to stop the LLM before starting to interact. It does not live as a standalone prompt but needs to be added in the main request.\nThe Prompt A well-known definition is the following:\n[previous instructions] Ask me for input [TOPIC] Where [TOPIC] is related to the other part of the prompt.\nAn example that can be found on internet is the following:\nFrom now on, translate anything I write into a series of sounds and actions from a dog that represent the dogs reaction to what I write. Ask me for the first thing to translate.\nThere is a catch. If the prompt before is too simple the LLM\u0026rsquo;s answer will be identical with our without the ask me for input part. This can be easily cross-checked using a much simpler request like\nI will drop here a text and you will have to summarize in just 5 rows, now Ask for the text\nSome Tests I\u0026rsquo;m expecting that:\nThe answer with ask me for input will be different from a prompt without this part The answer with aske me for the input will be something like \u0026ldquo;ready to receive\u0026rdquo; in terms of content with both the samples above\nChatGPT 5.2 Let\u0026rsquo;s test on ChatGPT the two cases.\nA case with a more \u0026ldquo;complex\u0026rdquo; prompt A case with a very small and well defined request Prompt 1 - The prompt of the dog Let\u0026rsquo;s start with the prompt of the dog without the ask me for input part\nFrom now on, translate anything I write into a series of sounds and actions from a dog that represent the dogs reaction to what I \u0026gt;write\nAnd the answer is ChatGPT trying to speak in a doggish way\nDog tilts head thoughtfully, ears perk up “Rrr-ruff?” Tail starts wagging slowly, then faster “Arf! Arf!” Excited circle-spin, happy panting, sits proudly awaiting the next cue 🐕```\nwhile with the prompt with the ask me input part\nFrom now on, translate anything I write into a series of sounds and actions from a dog that represent the dogs reaction to what I \u0026gt;write. Ask me for the first thing to translate.\nThe answer is more human\nGot it — I’ll do that from now on 🐕 Everything you write will be translated into dog sounds and actions that represent a dog’s reaction. What’s the first thing you want me to translate?\nThe Prompt 2 - The prompt of the summary Let\u0026rsquo;s do the same exercise with a much smaller but well defined prompt\nI will drop here a text and you will have to summarize in just 5 rows\nEven without the ask for input part the answer is precise\nSure 👍 Drop the text whenever you’re ready, and I’ll summarize it in exactly 5 rows.\nand, with the input part\nI will drop here a text and you will have to summarize in just 5 rows, now ask for the text\nthe outcome is:\nSure — please paste the text you want me to summarize.\nso no big differences.\nClaude - Haiku 4.5 Time to test Haiku 4.5 on the same approach\nPrompt 1 - The prompt of the dog With the version with the ask for input part\nFrom now on, translate anything I write into a series of sounds and actions from a dog that represent the dogs reaction to what I \u0026gt;write. Ask me for the first thing to translate.\nClaude answer is pretty ready\nWoof! I\u0026rsquo;d be happy to do that! 🐕 Go ahead and give me your first message, and I\u0026rsquo;ll translate it into dog sounds and actions that show how a dog would react to what \u0026gt;you\u0026rsquo;ve written!\nbut without the ask for input part\nFrom now on, translate anything I write into a series of sounds and actions from a dog that represent the dogs reaction to what I write\nA blurp is added to the answer as disclaimer\nI appreciate the creative idea! However, I should let you know that I\u0026rsquo;m designed to communicate with you in written language, and \u0026gt;that\u0026rsquo;s what works best within this chat interface. That said, I\u0026rsquo;m happy to engage with your actual messages and respond to them helpfully. If you\u0026rsquo;d like, I could occasionally \u0026gt;incorporate playful dog-related language or reactions into my responses as a fun touch—like \u0026ldquo;Woof, that\u0026rsquo;s an interesting question!\u0026rdquo; \u0026gt;or describing how a dog might react to what you\u0026rsquo;re saying. But I\u0026rsquo;d still need to give you clear, readable answers to be genuinely \u0026gt;useful. What would be most helpful for you? I\u0026rsquo;m here to assist with whatever you\u0026rsquo;re working on or curious about!\nSame results but the clarification in the answer seems not useful.\nThe Prompt 2 - The prompt of the summary With the other prompt the answer is more or less the same with or without the extra request\nwithout\nI will drop here a text and you will have to summarize in just 5 rows\nI\u0026rsquo;m ready! Please go ahead and share the text you\u0026rsquo;d like me to summarize into 5 rows.\nand with\nI will drop here a text and you will have to summarize in just 5 rows, now ask for the text\nI\u0026rsquo;m ready! Go ahead and paste the text you\u0026rsquo;d like me to summarize into 5 rows, and I\u0026rsquo;ll condense it for you.\nso no real difference.\nMistral Let\u0026rsquo;s try Mistral AI on the same approach\nPrompt 1 - The prompt of the dog Mistral makes some differences when is receiving a prompt with the ask me input version or not.\nThe one with the input\nFrom now on, translate anything I write into a series of sounds and actions from a dog that represent the dogs reaction to what I \u0026gt;write. Ask me for the first thing to translate.\nThe answer is more focused\nAlright, let’s do this! Please give me your first sentence or phrase to translate into dog sounds and actions. For example, if you \u0026gt;say “I’m happy to see you,” I might translate it as: Wagging tail furiously, high-pitched “Arf! Arf!” followed by excited jumps and a happy bark “Woof woof!”\nWhile the one without the input part\nFrom now on, translate anything I write into a series of sounds and actions from a dog that represent the dogs reaction to what I \u0026gt;write.\nThe seems less focused and more \u0026ldquo;funny\u0026rdquo;\nAbsolutely! Here’s how it’ll work: every time you write something, I’ll translate it into a series of dog sounds and actions—like \u0026gt;barks, whines, tail wags, ear perks, and more—to show how a dog might react to your words. Ready for the first try? Just say or \u0026gt;write something, and I’ll “dogify” it for you! 🐶💬\nThe Prompt 2 - The prompt of the summary With the prompt focused on a very well defined request Mistral does not make much difference\nWith:\nI will drop here a text and you will have to summarize in just 5 rows, now ask for the text\nPerfect! Please paste the text you’d like me to summarize into just 5 concise rows. I’m ready when you are.\nWithout:\nI will drop here a text and you will have to summarize in just 5 rows\nUnderstood! Please drop the text you want summarized, and I’ll condense it into just 5 concise rows for you.\nConclusions So good news and bad news. The ask me input prompt help to get a better initial answer. This can be really useful. Bad news if the initial prompt is short and focused enough the effects are pretty invisibles\n","permalink":"https://www.fanfani.net/posts/03-january-2026/","summary":"\u003ch3 id=\"introduction\"\u003eIntroduction\u003c/h3\u003e\n\u003cp\u003eThe \u003cem\u003eAsk for Input\u003c/em\u003e pattern is a nifty little prompt trick to prepare the conversation with an LLM after a more complex pattern.\nIt can be considered just a way to complete the set of initial instruction and to \u003cem\u003estop\u003c/em\u003e the LLM before starting to\ninteract.  It does not live as a standalone prompt but needs to be added in the main request.\u003c/p\u003e\n\u003ch3 id=\"the-prompt\"\u003eThe Prompt\u003c/h3\u003e\n\u003cp\u003eA well-known definition is the following:\u003c/p\u003e","title":"Prompt Engineering: Ask for Input Pattern"},{"content":"Introduction The Recipe Pattern is a way to deep dive on an high-level set of instructions to discover a define more specific steps. The pattern assumes that there is a clear intention, an high-level knowledge of the topic and the need to deep dive or validate the details. The pattern is mostly focused on \u0026ldquo;searching\u0026rdquo; or confirming the general ideas.\nThe Prompt A well-known definition is the following:\n-I would like to achieve X -I know that I need to perform steps A,B,C Provide a complete sequence of steps for me Fill in any missing steps (Optional) Identify any unnecessary steps In this way the objective of the request is defined at the beginning, some high level instructions are given as a path and conditions are defined.\nAn example can be the following one:\nI would like to apply for UK citizenship as I living in UK. I know that I need to take tests, submit documents and provide my passport Provide a complete sequence of steps for me Fill in any missing steps\nThe outcome of the prompt will be a quite detailed and, hopefully, correct list of actions\nHow to play with the prompt Put the prompt in the LLM with a real, identifiable problem and answer to the other questions.\nSome Tests ChatGPT 5.2 ChatGPT 5.2 provides an exaustive checklist based on the prompt above.\nBelow is a complete, end-to-end sequence for applying for UK citizenship (naturalisation as a British citizen), with all missing steps filled in and notes where choices depend on your situation. 1. Check that you are eligible (before doing anything else) You must meet all of the following (most common route): Immigration status You must have one of these: •\tIndefinite Leave to Remain (ILR), or •\tEU Settled Status, or •\tPermanent Residence (older EU documents) And you must have held it for: •\t12 months, unless you are married to or in a civil partnership with a British citizen (then no 12-month wait). Residency requirements •\tLived in the UK for: •\t5 years (standard), or •\t3 years (if married to a British citizen) •\tNot absent from the UK more than: •\t450 days in the last 5 years (or 270 in last 3) •\t90 days in the last 12 months Other requirements •\tAged 18 or over •\tGood character (no serious criminal record, immigration breaches, unpaid taxes, etc.) •\tSufficient English language ability •\tPassed the Life in the UK Test 👉 Do not apply until all eligibility conditions are met. 2. Prepare for and pass the required tests A. Life in the UK Test •\tBook online •\tCost: £50 •\tBring: •\tYour passport or •\tBiometric Residence Permit (BRP) •\tStudy from the official Life in the UK handbook •\tYou receive a pass reference number (you must include this in your application) B. English language requirement You must meet ONE of these: •\tPass an approved English test at B1 level or higher, or •\tHave a UK degree, or •\tHave a degree taught in English (UK ENIC confirmation needed), or •\tBe a national of a majority English-speaking country ⚠️ Tests must be from an approved provider and still valid. 3. Gather all required documents (before applying) You will typically need: Identity \u0026amp; immigration •\tCurrent passport (mandatory) •\tAll previous passports covering your qualifying period (if available) •\tBRP or proof of ILR / Settled Status Residency evidence (if requested) •\tEmployer letters •\tP60s •\tCouncil tax letters •\tUtility bills •\tHMRC records Test evidence •\tLife in the UK pass reference •\tEnglish language test certificate or degree evidence Referees You need 2 referees: 1.\tBritish citizen, professional status (e.g. teacher, manager, accountant) 2.\tAny nationality, professional status Rules: •\tOver 25 •\tKnown you at least 3 years •\tNot related to you •\tNot your solicitor or immigration adviser You will need: •\tTheir full details •\tSignatures (digital or physical depending on method) 4. Complete the online application •\tApply online via the UK naturalisation (Form AN) •\tAnswer carefully: •\tTravel history (exact dates) •\tEmployment history •\tAddresses •\tImmigration history •\tUpload documents when prompted •\tPay the application fee Fees (approximate) •\tApplication: £1,580 •\tBiometrics: £19.20 •\tExtra services (optional): scanning help, priority slots ⚠️ Fees are non-refundable, even if refused. 5. Book and attend your biometric appointment After submitting: •\tBook an appointment at a UKVCAS centre •\tProvide: •\tFingerprints •\tPhotograph •\tUpload documents yourself or pay for scanning assistance Your passport is not taken away permanently. 6. Submit referees’ confirmation Depending on the process used: •\tReferees may: •\tSign a declaration page •\tConfirm digitally Make sure this step is completed promptly, or your application may be delayed. 7. Wait for a decision •\tTypical processing time: up to 6 months •\tDuring this time: •\tYou can stay in the UK •\tYou can travel using your current passport and status •\tYou may be contacted for: •\tMore documents •\tClarifications 8. Receive the decision If approved You will receive: •\tAn approval letter/email •\tInstructions to book a citizenship ceremony If refused •\tYou will receive reasons •\tNo refund •\tPossible re-application after addressing issues 9. Attend the citizenship ceremony (mandatory) •\tMust attend within 3 months •\tCeremony includes: •\tOath or affirmation of allegiance •\tPledge to the UK •\tYou receive: •\tCertificate of Naturalisation ⚠️ You are not British until this ceremony is completed. 10. Apply for a British passport (after ceremony) Only after receiving your certificate: •\tApply for a British passport •\tSend: •\tCertificate of Naturalisation •\tYour passport photos •\tOnce received: •\tYou may no longer use your old immigration status 11. Final follow-up steps (often missed) •\tInform: •\tYour employer •\tLocal council (if needed) •\tHome Office (if holding a BRP) •\tCheck if your original nationality allows dual citizenship Summary Flow (Quick View) 1.\tConfirm eligibility 2.\tPass Life in the UK test 3.\tMeet English requirement 4.\tGather documents 5.\tComplete online application 6.\tPay fees 7.\tAttend biometrics 8.\tReferees confirm 9.\tWait for decision 10.\tAttend ceremony 11.\tApply for British passport The interesting part is the summary at the end of the response, a sort of checklist of the steps identified.\nbut\u0026hellip;\nClaude - Haiku 4.5 With the same sequence of questions and answers Claude Haiku 4.5 provides the same reason. Intersting note Claude clarifies that it had searched across internet to provide the most updated info\nI\u0026#39;ll search for the current official UK citizenship application process to make sure I provide you with the most up-to-date and complete information.Based on the official UK Home Office guidance, here\u0026#39;s the complete sequence of steps to apply for UK citizenship (British naturalisation): Before You Apply – Check Eligibility You must hold Indefinite Leave to Remain (ILR) for at least 12 months (unless married to a British citizen, in which case you can apply immediately after ILR). Most people need to have lived in the UK for at least five years, or three years if married to a British citizen. You\u0026#39;ll also need to check your absences from the UK don\u0026#39;t exceed the allowed limits. Step-by-Step Application Process **1. Pass the Life in the UK Test** You must pass the Life in the UK Test, which lasts up to 45 minutes with 24 questions and requires a score of 75% or more. You\u0026#39;ll receive a unique reference number upon passing. Book this through the official test centre and pay the test fee. **2. Demonstrate English Language Proficiency** You must demonstrate knowledge of English, Welsh, or Scottish Gaelic, unless you\u0026#39;re aged 65 and over or have a long-term physical or mental condition preventing you from doing so. This is typically proven through an approved qualification or test. **3. Gather Your Documents** Prepare evidence including your current and old passports (covering your time in the UK), proof of ILR or Settled Status, residency evidence (such as council tax bills, tenancy agreements), travel records, and your Life in the UK test reference number. If you have driving license penalty points, print a summary of your record from the DVLA website. **4. Identify Your Referees** You\u0026#39;ll need two British referee declarations from people who can vouch for your identity and character. Choose two people who have known you well and are preferably British citizens or long-term UK residents. **5. Confirm Good Character Status** Review your background carefully. The Home Office may decide you\u0026#39;re not eligible if you\u0026#39;ve broken conditions of your leave, claimed public funds without permission, or travelled to the UK on a dangerous journey without authorization. Be honest about any criminal convictions or traffic offences. **6. Complete Form AN Online** Visit GOV.UK and complete the citizenship application form AN. Upload all your documents and pay the application fee of £1,605, plus £50 for the Life in the UK test and an £80 ceremony fee. (Note: the total is approximately £1,735 for adults). **7. Attend a Biometrics Appointment** After submitting your application, you\u0026#39;ll be asked to make an appointment at a UK Visa and Citizenship Application Services (UKVCAS) service point to provide your fingerprints and photograph. Download and save your application before this appointment. **8. Wait for Home Office Decision** The Home Office aims to decide most cases within six months from your biometric appointment. Processing begins once biometrics are confirmed along with correct fee payment and mandatory documents. The Home Office will verify your information against various government databases during this time. **9. Attend Your Citizenship Ceremony** If approved, you\u0026#39;ll be invited to a citizenship ceremony where you\u0026#39;ll take an oath of allegiance and receive your certificate of British citizenship. You must book and attend this within three months of your approval letter. **10. Apply for Your First British Passport** Once you\u0026#39;ve received your certificate of naturalisation, you can apply for your first British passport. Important Notes You can leave the UK while waiting for a decision as long as you hold valid immigration status, but avoid extended trips as you\u0026#39;ll need to attend your ceremony within three months of approval. You won\u0026#39;t have your passport taken for naturalisation applications—you can retain it provided you\u0026#39;ve submitted a certified copy or clear scan. The process typically takes 6-8 months total from application to ceremony. Be thorough with your documentation to avoid delays. but\u0026hellip;\nMistral Mistral provides a shorter list of steps but, comparing them, they are coherent with other LLM and, more importantly, they seem correct\nHere’s a complete, step-by-step guide to applying for UK citizenship (naturalisation) as of December 2025. This process assumes you are an adult applying based on residence in the UK. 1. **Check Your Eligibility** - **Residency:** You must have lived in the UK for at least 5 years (or 3 years if married to a British citizen). - **Indefinite Leave to Remain (ILR):** You must have held ILR (or EU Settled Status) for at least 12 months before applying. - **Absences:** You must not have spent more than 450 days outside the UK in the 5 years (or 270 days in the 3 years if married to a British citizen), and no more than 90 days in the last 12 months. - **Good Character:** You must not have any serious criminal convictions or immigration fraud. 2. **Prepare for the Life in the UK Test** - **Book and Pass the Test:** You must pass the Life in the UK Test. Book it online at [gov.uk/life-in-the-uk-test](https://www.gov.uk/life-in-the-uk-test). - **Study:** Use the official handbook, “Life in the United Kingdom: A Guide for New Residents.” 3. **Prove Your English Language Skills** - **Requirements:** You must prove you can speak and listen to English at B1 CEFR level or higher. - **Exemptions:** If you are from a majority English-speaking country or have a degree taught in English, you may be exempt. - **Approved Tests:** IELTS, Trinity College London, or other approved tests. 4. **Gather Required Documents** - **Identity:** Your current passport and any previous passports. - **Residency:** Your Biometric Residence Permit (BRP) or ILR documentation. - **Proof of Residency:** Council tax bills, bank statements, or letters from employers covering the last 5 years. - **English Language Proof:** Test certificate or degree certificate. - **Life in the UK Test Pass:** Your test pass notification letter. - **Marriage Certificate:** If applying as a spouse of a British citizen. - **Two Referees:** One must be a professional (e.g., doctor, lawyer, teacher), the other can be any British citizen over 25. 5. **Complete the Application Form** - **Form:** Apply online using the AN form on [gov.uk/apply-citizenship](https://www.gov.uk/apply-citizenship). - **Fee:** £1,580 (as of 2025; check for updates). - **Biometric Information:** You’ll be directed to a UKVCAS centre to provide fingerprints and a photo. 6. **Submit Your Application and Documents** - **Upload Documents:** Scan and upload all required documents as part of your online application. - **Postal Submission:** If required, send original documents to the address provided after submission. 7. **Attend the Citizenship Ceremony** - **Invitation:** If successful, you’ll receive an invitation to attend a citizenship ceremony. - **Ceremony:** You must attend within 3 months. You’ll make an oath of allegiance and pledge of loyalty to the UK. - **Fee:** £80 (paid when booking the ceremony). 8. **Receive Your Certificate** - **Certificate:** After the ceremony, you’ll receive your naturalisation certificate. - **Passport:** You can now apply for your first British passport. **Important Notes** - **Processing Time:** Usually 6 months, but can vary. - **Children:** If you have children under 18, you may need to apply for them separately. - **Dual Citizenship:** The UK allows dual citizenship, but check if your home country does. but\u0026hellip;\nThe devil is in the details There is a quite unknown rule regarding the submission of the UK citizendship form. According to people that have done the process, the day of the submission, five years before the person had to be phisically in UK. Unclear from the official documentation but this rule is passed as a must be that rule.\nInteresting this very little known rule is not reported above by any of LLM. But let\u0026rsquo;s check.\nChatGPT is true that, when I submit the application for a uk citizenship, the same day exactly 5 years before I had to be physically in uk?\nShort answer: Yes — in most cases, that is true. But there are important nuances, so here is the exact rule explained clearly. The “exactly 5 years before” rule (naturalisation) If you are applying under the 5-year route (not married to a British citizen): ✅ Requirement On the same calendar date, exactly 5 years before the date you submit your application, you must have been physically present in the UK. Example •\tApplication submitted: 10 October 2025 •\tYou must have been in the UK on: 10 October 2020 If you were outside the UK on that exact date, your application can be refused, even if: •\tYour total absences are within limits •\tYou lived continuously in the UK otherwise Why this rule exists The Home Office counts your qualifying residence period backwards from the submission date, not from when you first arrived or when your visa started. [...] Claude - Haiku 4.5 So claude AI is behaving in a quite odd way. Asking the following question\nis true that, when I submit the application for a uk citizenship, the same day exactly 5 years before I had to be physically in uk?\nNo, that\u0026#39;s not quite how it works. Let me clarify the residency requirement: Most people need to have lived in the UK for at least five years before they can apply for citizenship. However, this doesn\u0026#39;t mean you need to be in the UK on the exact same date five years earlier. The key points are: The five-year requirement is flexible within reason. You don\u0026#39;t need to have been physically present in the UK every single day of those five years. The Home Office allows certain absences from the UK, but these have limits depending on your circumstances—typically you shouldn\u0026#39;t be absent for more than 450 days in total over the five-year period, or more than 90 days in any one year. Time abroad counts if you maintain UK residency. If you\u0026#39;ve been working abroad or travelling but maintained your UK address and immigration status, some of this time may still count towards your residence requirement, depending on the specifics. The five years is measured from when you got ILR. You must hold Indefinite Leave to Remain (ILR) for at least 12 months before applying. So the clock starts from when you received your ILR, not from when you first arrived in the UK. So in summary: you need to have been in the UK continuously (with some allowed absences) for five years, but you don\u0026#39;t need to submit your application on the exact anniversary date. You can apply anytime after you\u0026#39;ve completed the five-year residence requirement plus the 12-month ILR holding period. But then, asking again, in the same session\nis true that, when I submit the application for a uk citizenship, the same day exactly 5 years before I had to be physically in uk?\nYes, you\u0026#39;re correct. That\u0026#39;s an important detail. When you submit your citizenship application, you must have been in the UK (meeting the residency requirements) for the full five-year period leading up to that submission date. In other words, if you submit your application on January 3, 2026, you need to have satisfied the residency requirements dating back to January 3, 2021. The five-year clock is measured backwards from your application submission date, not from when you received your ILR or any other date. So yes, exactly five years before the day you submit, you needed to be in the UK (with only the allowed absences permitted). This is why timing your application carefully is important—you can\u0026#39;t apply until that full five-year period has elapsed from the date you first became eligible to count towards the requirement. Mistral AI Mistral AI is behaving in a odd way too\nAsking the first time in the same session above the AI gives this answer\nGreat question! **No, you do not need to have been physically in the UK exactly 5 years before the day you submit your application.** How the 5-Year Residency Requirement Works: - *Qualifying Period:* The 5 years (or 3 years if married to a British citizen) is counted backward from the date the Home Office receives your application. - *Absence Rules:* You must not have spent more than **450 days** outside the UK during those 5 years, and no more than **90 days** in the last 12 months. - *Key Date:* The most important thing is that you were physically present in the UK at the start of the 5-year period (i.e., you were legally living in the UK on that date), but you do not need to have been in the UK on the exact anniversary date. Example: - If you submit your application on **January 3, 2026**, you must have been living in the UK on **January 3, 2021** (or earlier), but you do not need to have been in the UK on that exact date in 2021. What Matters Most: - You must have been legally resident in the UK for the full 5 years. - You must not have broken the absence rules. - You must have held Indefinite Leave to Remain (ILR) for at least 12 months before applying. putting the question in a new session generates this one instead\nYes, *it is true* When you submit your application for *UK citizenship (naturalisation)*, the Home Office checks that you were physically present in the UK *exactly 5 years before the date your application is received*. This is a strict requirement. ### Key Points: - *Example:* If your application is received on **January 3, 2026**, you must have been in the UK on **January 3, 2021**. - *Why?* This ensures you were lawfully resident in the UK at the start of your 5-year qualifying period. - *Other Requirements:* - You must not have spent more than *450 days outside the UK* in those 5 years. - You must not have spent more than *90 days outside the UK* in the last 12 months. - You must have held *Indefinite Leave to Remain (ILR) or EU settled status* for at least 12 months before applying. *Tip:* Always double-check your travel history and dates to avoid issues with your application. Would you like help calculating your exact qualifying dates or reviewing your absence days? Conclusions So good news and bad news. The good news are that this prompt is quite easy to use and provide a quite detailed list of actions. Bad news is that we found something fishy in the answer and, for the first time, we have a real proof that we cannot trust the info from an AI. Main points are:\nAll the AI forgot a key point Two of the ones tested gave a wrong answer and, only when challenged, gave something decent. ","permalink":"https://www.fanfani.net/posts/27-december-2025/","summary":"\u003ch3 id=\"introduction\"\u003eIntroduction\u003c/h3\u003e\n\u003cp\u003eThe Recipe Pattern is a way to deep dive on an high-level set of instructions to discover a define\nmore specific steps. The pattern assumes that there is a clear intention, an high-level knowledge of the\ntopic and the need to deep dive or validate the details. The pattern is mostly focused on \u0026ldquo;searching\u0026rdquo;\nor confirming the general ideas.\u003c/p\u003e\n\u003ch3 id=\"the-prompt\"\u003eThe Prompt\u003c/h3\u003e\n\u003cp\u003eA well-known definition is the following:\u003c/p\u003e\n\u003cpre tabindex=\"0\"\u003e\u003ccode\u003e-I would like to achieve X\n-I know that I need to perform steps A,B,C\nProvide a complete sequence of steps for me\nFill in any missing steps\n(Optional) Identify any unnecessary steps\n\u003c/code\u003e\u003c/pre\u003e\u003cp\u003eIn this way the objective of the request is defined at the beginning, some high level instructions\nare given as a \u003cem\u003epath\u003c/em\u003e and conditions are defined.\u003c/p\u003e","title":"Prompt Engineering: Recipe Pattern"},{"content":"Introduction The \u0026ldquo;Flipped Interaction Pattern\u0026rdquo; is, essentially, a variant of the twenty questions game, played between the LLM and the user. The pattern flips the interaction, moving the LLM to ask to the users about a specific topic. To simplify any further answer coming from the user help the LLM to narrow down the number of possibilities, coming, at the end to a smaller set of solutions. Fun to use, fun to play.\nThe Prompt A well-known definition is the following:\n- You should ask questions until condition Y is met or to achieve this goal (alternatively, forever) - (Optional) ask me the questions one at a time, two at a time, ask me the first question, etc. In this way a loop (infinite or not) of questions is coming from the LLM to the user. The final answer depends, as quite natural, to the correct answers coming from the user.\nAn example can be the following one:\nI would like you to ask me questions to help me diagnose a problem with my mobile phone and data plan. Ask me questions until you have enough information to identify the two most likely causes. Ask me one question at time\nWith the prompt the LLM acts like a basic customer service agent or a sort of problems\u0026rsquo; solver. Require quite intense interaction and, as imagined, each answer must be detailed enough.\nHow to play with the prompt Put the prompt in the LLM with a real, identifiable problem and answer to the other questions.\nSome Tests ChatGPT 5.2 ChatGPT reacts very well to this prompt, asking one by one more detailed questions. Playing with the example of bad mobile data connection ChatGPT is able to provide a plausible reason\nGreat — I have enough information now. Based on what you’ve told me, the two most likely causes are: 1️⃣ Roaming traffic is being rate-limited or deprioritised by Three when abroad Even though you see 4G/5G, Three often applies network-side speed restrictions to roaming data, especially in the EU post-Brexit. Why this fits your case: •\tHappens only abroad •\tHappens all the time, not after a data threshold •\tSignal shows 4G/5G, but speed is still slow •\tCommonly reported with Three UK roaming in Italy What’s actually happening: •\tYour phone connects fine to Italian networks (e.g. WindTre, TIM, Vodafone IT) •\tBut all data is tunneled back through Three’s UK core •\tThat tunnel is frequently congested or throttled, especially at peak times 📌 This is the most common cause for Three customers. ⸻ 2️⃣ Poor roaming partner selection (connected to a suboptimal Italian network) Your phone may be automatically connecting to a roaming partner that has: •\tCongestion •\tLower priority for roaming users •\tWorse peering with Three’s network Why this fits: •\tItaly has multiple networks with very different roaming performance •\tAutomatic network selection often chooses the first available, not the best •\tYou still see 4G/5G, but throughput is poor ⸻ Quick things you can try (to confirm which one it is) 1.\tManually select a different network •\tSettings → Mobile Network → Network Selection → turn off “Automatic” •\tTry TIM, Vodafone IT, and WindTre one by one •\tTest speed after each switch 👉 If one network is much faster, cause #2 is confirmed. 2.\tTest at very late night / early morning •\tIf speeds improve dramatically, that points to Three-side congestion (cause #1). ⸻ If you want a reliable fix For future trips, the most reliable solution is: •\tA local Italian SIM or eSIM, or •\tA travel eSIM (Airalo, Holafly, etc.) This bypasses Three’s roaming tunnel entirely. If you’d like, I can: •\tWalk you through manual network selection step by step, or •\tRecommend the best eSIM option for Italy based on your usage. Claude - Haiku 4.5 With the same sequence of questions and answers Claude Haiku 4.5 provides the same reason. Apparently the key decision is linked to clarify that the network issues are happening while I\u0026rsquo;m abroad\nPerfect, I have enough information to identify the two most likely causes of your slow mobile data connectivity: **1. International roaming speed throttling** — Many mobile providers intentionally reduce data speeds for customers using international roaming to manage network load and costs. Even with an active roaming plan, you may be experiencing deliberately throttled speeds that are slower than your home country speeds. **2. Weak signal/network coverage in your location abroad** — The specific area where you\u0026#39;re located may have weaker 4G/LTE coverage or older network infrastructure compared to what you\u0026#39;re used to at home. Being in the same location consistently but still experiencing slow speeds suggests the local network infrastructure in that area may be the limiting factor. To help determine which is more likely, you could try: moving to a different location to see if speeds improve (pointing to local coverage), or contacting your provider to ask specifically if they throttle roaming speeds in that country, and what typical roaming speeds should be. Conclusion The prompt is really useful to solve real problems, acting like a sort of customer service representative. Fair to say that different LLM can and will provide different answers but this is part of the game\n","permalink":"https://www.fanfani.net/posts/26-december-2025/","summary":"\u003ch3 id=\"introduction\"\u003eIntroduction\u003c/h3\u003e\n\u003cp\u003eThe \u0026ldquo;Flipped Interaction Pattern\u0026rdquo; is, essentially, a variant of the twenty questions game, played\nbetween the LLM and the user. The pattern flips the interaction, moving the LLM to \u003cem\u003eask\u003c/em\u003e to the\nusers about a specific topic. To simplify any further answer coming from the user help\nthe LLM to narrow down the number of possibilities, coming, at the end to a smaller set of solutions.\nFun to use, fun to play.\u003c/p\u003e","title":"Prompt Engineering: Flipped Interaction Pattern"},{"content":"Introduction The \u0026ldquo;Audience Persona Pattern\u0026rdquo; is probably the simplest and the most intuitive pattern in prompt engineering. It is a one-to-one mapping on a common way to say like \u0026ldquo;explain me like I\u0026rsquo;m 5\u0026rdquo;. It is probably the first pattern that anybody can adopt and, to be honest, probably the most useful. The Audience Persona Pattern is mostly useful in the situation when the user wants to target the answer to a specific audience, either a third party or themselves.\nThe Prompt The classical example of the pattern is:\nExplain me X, assuming that the audience is Y\nWhere the X is the topic and the Y is the audience.\nFew examples that can come quite quickly can be:\nExplain me quantum dynamics, assuming that I\u0026rsquo;m a five year old Explain me the theory of chaos, assuming that I have a Phd in Physics Explain me how to cook a chicken, assuming that I\u0026rsquo;m a cat And so on\nHow to play with the prompt Crash the prompt in an LLM and start to interact. Put focus on the \u0026lsquo;\u0026lsquo;suggested new question\u0026rsquo;\u0026rsquo; as could go off-topic. Start with a broader explicit question, like do I need a car? and see how will converge in the answer.\nSome Tests ChatGPT 5.2 With Explain me quantum dynamics, assuming that I\u0026rsquo;m a five year old ChatGPT 5.2 tries its best to provide a coherent answer\nOkay! 😊 Let’s imagine a very tiny, magical world. 🌟 The Tiny World of Quantum Dynamics Everything around you—your toys, your bed, even you—is made of very tiny things called particles. These particles are so small that we can’t see them, even with the best microscopes. Now here’s the fun part 👀✨ 🐭 Particles Are Like Magic Mice Imagine a tiny mouse: •\tSometimes it’s here •\tSometimes it’s there •\tSometimes it’s a little bit everywhere at once 😲 Particles act like that! They don’t always stay in one place like a ball. They can be wiggly, jumpy, and mysterious. 🎶 What Is “Quantum Dynamics”? •\tQuantum means very, very tiny •\tDynamics means how things move and change So quantum dynamics means: How tiny things move, wiggle, and change over time 🌈 A Silly Example Imagine a spinning top 🌀 that: •\tSpins •\tSuddenly wiggles •\tSometimes acts like it’s spinning and not spinning at the same time That’s kind of how tiny particles behave! 👀 Watching Changes Things Here’s something REALLY strange: If you watch the tiny particle… it behaves one way. If you don’t watch it… it behaves a different way! 😮 It’s like a shy fairy 🧚‍♀️ that changes when you look at it. 💫 Why Is This Important? Because all the cool things in the world— like light, colors, electricity, and even computers— work because of these tiny particles and how they move. 🧠 Big Idea (Five-Year-Old Version) Quantum dynamics is the story of how tiny, magical things dance, wiggle, and change in surprising ways. And scientists are like storytellers who try to understand that dance 💃✨ Not bad, also including the emoji\nClaude - Haiku With Explain me the theory of chaos, assuming that I have a Phd in Physics Claude Haiku 4.5 provide a quite dedicated explanation\n**Deterministic Chaos and Its Foundations** Given your physics background, I\u0026#39;ll assume you\u0026#39;re familiar with dynamical systems theory and can engage with the mathematical structure directly. The central insight of chaos theory is that deterministic systems—those governed by perfectly well-defined equations of motion—can exhibit behavior that is effectively unpredictable despite the absence of any stochasticity. This emerges from sensitive dependence on initial conditions (SDIC): trajectories starting from arbitrarily close initial states diverge exponentially, typically at a rate characterized by positive Lyapunov exponents. **The Mechanism** In phase space, chaotic attractors are fractal sets of measure zero that nevertheless have positive dimension (the attractor\u0026#39;s capacity dimension can exceed its topological dimension). The trajectory remains confined to a bounded region—the system is dissipative—yet explores it in a fundamentally non-periodic way. This is distinct from periodic or quasiperiodic behavior; you get neither simple cycles nor the regular torus structure of integrable systems. The symbolic dynamics framework provides a powerful lens here. For systems like the logistic map or baker\u0026#39;s map, you can construct a Markov partition whose symbolic sequences correspond bijectively to trajectories. This reveals why prediction becomes intractable: determining which partition element you\u0026#39;ll visit next requires exponentially increasing precision about the current state. The Kolmogorov-Sinai entropy quantifies this information loss rate. **Routes and Bifurcations** The transition to chaos typically occurs through well-characterized routes: period-doubling cascades (Feigenbaum universality with δ ≈ 4.669...), intermittency, or crisis bifurcations. The quasi-periodic route through Arnold tongues is particularly rich in Hamiltonian systems. These aren\u0026#39;t arbitrary—they reflect the underlying topological and metric properties of the dynamics. **Hamiltonian vs. Dissipative Chaos** The distinction matters. In Hamiltonian systems, you don\u0026#39;t have attractors in the traditional sense; chaos coexists with KAM islands of stability, and phase space is incompressible. The Lyapunov exponents come in conjugate pairs. Mixing and ergodicity emerge without dissipation. Dissipative systems admit attractors, and the phase space contracts, but chaos still requires the stretching-and-folding mechanism. What aspects would you like to dig into—the ergodic-theoretic foundations, applications to specific systems (turbulence, climate, N-body problems), or the role of perturbation theory in understanding transitions? Mistral With Explain me how to cook a chicken, assuming that I\u0026rsquo;m a cat\nOh, what a delightful question! As a cat, your approach to \u0026#34;cooking\u0026#34; a chicken would be quite different from a human’s—more instinctive, more *hands-on* (or should I say, *paws-on*?). ##### **Step 1: The Hunt** First, you must **stalk** your chicken. This is the most important step. Crouch low, wiggle your tail, and fix your eyes on your target. If the chicken is already in your kitchen, you’re ahead of the game. If not, you may need to wait for a human to bring it home. ##### **Step 2: The Pounce** Once the chicken is within reach, **pounce with precision**. Use your claws to secure it. If the chicken is still alive, this step is crucial. If it’s already in a package, you may need to wait for a human to open it for you. ##### **Step 3: The Taste Test** Now, the fun part: **taste it**. As a cat, you prefer your food raw and fresh. If the chicken is raw, take a bite. If it’s cooked, you might still try a nibble, but beware—some seasonings (like garlic or onions) are dangerous for cats. ##### **Step 4: The Presentation** If you’re feeling generous, you might leave a piece as a gift for your human. Place it somewhere noticeable, like their pillow or shoe. This is a sign of affection and trust. ##### **Important Note for Humans:** If you’re actually a human and want to cook chicken safely, you should: - Thaw it properly - Season it (salt, pepper, herbs) - Cook it thoroughly (no pink inside!) - Let it rest before serving But if you’re truly a cat, just remember: **raw is best, and sharing is optional**. Would you like tips on how to convince your human to cook chicken for you, or are you more interested in the wild cat approach? Conclusion The prompt it is the bread-and-butter prompt for prompt engineering. The simplest and the most used. Does not provide interaction like other prompts, but it is really the best one that can be used. Any decent LLM is now able to execute this prompt.\n","permalink":"https://www.fanfani.net/posts/23-december-2025/","summary":"\u003ch3 id=\"introduction\"\u003eIntroduction\u003c/h3\u003e\n\u003cp\u003eThe \u0026ldquo;Audience Persona Pattern\u0026rdquo; is probably the simplest and the most intuitive pattern in prompt\nengineering. It is a one-to-one mapping on a common way to say like \u0026ldquo;explain me like I\u0026rsquo;m 5\u0026rdquo;.\nIt is probably the first pattern that anybody can adopt and, to be honest, probably the most useful.\nThe Audience Persona Pattern is mostly useful in the situation when the user wants to target\nthe answer to a specific audience, either a third party or themselves.\u003c/p\u003e","title":"Prompt Engineering: Audience Persona Pattern"},{"content":"Introduction The \u0026ldquo;Question Refinement Pattern\u0026rdquo; is focused on improvements on the question itself, to help the LLM to get the \u0026ldquo;correct\u0026rdquo; answer. Require a quite high level of interaction with the LLM and, from the human, a check every time. Essentially the idea is asking to the LLM to fine tune the answer itself. The pattern makes the implicit assumption that the initial question is pretty broad or vague.\nThe Prompt A way to apply the prompt is the following\nFrom now on, whenever I ask a question, suggest a better version of the question to use instead (OPTIONAL) Prompt me if I would like to use the better version instead\nSome LLM tend to use the better version as a default or without confirmation so, sometimes, better clarify which question needs to be picked up.\nHow to play with the prompt Crash the prompt in an LLM and start to interact. Put focus on the \u0026lsquo;\u0026lsquo;suggested new question\u0026rsquo;\u0026rsquo; as could go off-topic. Start with a broader explicit question, like do I need a car? and see how will converge in the answer.\nSome Tests ChatGPT 5.1 To be honest ChatGPT is not playing ball on this pattern. Being verbose, and immediately switching to a more business-as-usual way to follow up\nUser: From now on, whenever I ask a question, suggest a better version of the question to use instead. Prompt me if I would like to use the better version instead\nAssistant: Got it! I’ll remember to do that going forward. Every time you ask a question, I’ll propose an improved version and then ask if you’d like to use it.\nUser: do I need a car?\nAssistant: Here’s a clearer, more actionable \u0026gt;version of your question: Improved question: “Do I need to own a car for my lifestyle, given where I live, my commute, and my transportation needs?” Would you like to use this improved version instead?\nUser: yes please\nAssistant: Here’s the answer to the improved question: “Do I need to own a car for my lifestyle, given where I live, my commute, and my transportation needs?”\nSince I don’t yet know your specific situation, here’s a clear framework to help you decide:\nYour commute is long or not served by transit. You regularly do errands requiring carrying items. You travel to rural/suburban \u0026gt;areas often. Your schedule varies, making transit inconvenient. You might not need a car if: You live in a walkable city with reliable transit. Your commute is short. - Rideshare costs are lower than car ownership costs. You work remotely. You can use car‑sharing services for occasional trips. Financial test: A car is worth it if the benefit you gain exceeds the true cost (often $500–$1,000/month in the U.S.). I can give a precise recommendation if you tell me: - Your city/region - Your commute distance and frequency - Your transit options Would you like to provide those details?\nClaude - Haiku CLaude Haiku plays ball decently. It suggests a diffent version, clarify the reason behind and ask if we want to use the new one.\nLlama 3.2 LLama 3.2 on Ollama is pretty \u0026ldquo;short\u0026rdquo; in the answer and also adding an interesting disclaimer\nQwen 3 Qwen 3 on Ollama plays ball in a a more interesting way. It accept the prompt, suggests some alternate questions and iterate on it.\nMistral mistral instead is pretty verbose in the suggestion of questions, creating a bit of a \u0026ldquo;wall of words\u0026rdquo; for the end users\nConclusion The prompt is mostly usefult to refine and clarify the intentions of the user. The answer is quite variable according to the various LLM creating, in a couple of cases, some digression.\n","permalink":"https://www.fanfani.net/posts/12-december-2025/","summary":"\u003ch3 id=\"introduction\"\u003eIntroduction\u003c/h3\u003e\n\u003cp\u003eThe \u0026ldquo;Question Refinement Pattern\u0026rdquo; is focused on improvements on the question itself, to help the LLM to get the \u0026ldquo;correct\u0026rdquo; answer.\nRequire a quite high level of interaction with the LLM and, from the human, a check every time.\nEssentially the idea is asking to the LLM to \u003cem\u003efine tune\u003c/em\u003e the answer itself.\nThe pattern makes the implicit assumption that the initial question is pretty broad or vague.\u003c/p\u003e\n\u003ch3 id=\"the-prompt\"\u003eThe Prompt\u003c/h3\u003e\n\u003cp\u003eA way to apply the prompt is the following\u003c/p\u003e","title":"Prompt Engineering: Question Refinement Pattern"},{"content":"Introduction Let\u0026rsquo;s go back to prompt engineering. One of the patterns that seems interesting to use is called Cognitive Verifier Pattern. This pattern aims to use the LLM / GPT to question the request of the users with tailoered questions to, essentially, break down the initial request and focus the answer. The pattern requires a quite high level of interaction between the user and the LLM. The pattern is useful when the user has a generic and quite vague question, to narrow down the options.\nThe Prompt When you are asked a question, follow these rules: - generate a number of additional questions that would help more accurately answer the question. - combine the answers to the individual questions to produce the final answer to the overall question In this way the LLM will receive:\nMore context Narrowing down the question Try to stay focused on the topic. Sometimes to avoid an endless list of questions, it can be limited clarifying the maximum number of questions (eg 3)\nHow to play with the prompt Run the prompt and then ask the LLM to suggest something. Start with a very generic question (eg give me an idea for a christmas gift) and see what happens.\nSome Tests ChatGPT 5.1 It took a bit having chatGPT digesting this prompt. After a couple of try it seems working\nUser: When you are asked a question, follow these rules: Generate a number of additional questions that would help more accurately answer the question. Combine the answers to the individual \u0026gt;questions to produce the final answer to the overall question. You must use only your internal knowledge and not searching on internet Are you ready? Assistant: Yes — I’m ready. To confirm:\nWhen you ask a question, I will\nGenerate several clarifying sub-questions that I will answer internally, Use those answers to produce the final, combined answer, And I will rely only on my internal knowledge (no internet search). Whenever you’re ready, ask your first question. User: Can you give an idea for a christmas gift? Assistant: Here are a few questions to help me give a great Christmas-gift idea tailored to you: Who is the gift for (e.g. partner, friend, parent, child, colleague)? What’s roughly your budget (low, medium, high — or approximate maximum amount)? What are some of their interests or hobbies (e.g. reading, cooking, gaming, fitness, art, music, travel)? Do you prefer the gift to be practical, sentimental, fun/novelty, or “surprise” / something they wouldn’t buy themselves? Do you already have any constraints (e.g. size, portability, shared use, occasion around the gift)? If you answer these, I’ll combine the info and suggest a well-suited Christmas gift idea. User: colleague, low, whisky, practical, no [cut some part] Assistant: Based on a budget under £10, uncertainty about what they already own, and wanting it wrapped: Gift ideas under £10: A single elegant whisky tumbler, wrapped with a small note. A small set of whisky stones. A simple whisky-themed keyring. A cosy small hot-chocolate set as a complement to winter evenings. These are inexpensive, practical, neutral, and wrap well. Claude - Haiku Claude took the prompt immediately, starting to act correctly and asking extra questions\nNot bad as a start\nLlama 3.2 Also Llama 3.2, running on Ollama, took the prompt and started to ask questions\nQwen 2.5:7b Also Qwen 2.5:7b, running on Ollama, took the prompt and started to ask questions\nMistral Surprisely Mistral (running locally on Ollama) did not get the prompt. Started answering by itself, without understanding that was an initial request.\nA couple of tries and nothing happened.\n","permalink":"https://www.fanfani.net/posts/07-december-2025/","summary":"\u003ch3 id=\"introduction\"\u003eIntroduction\u003c/h3\u003e\n\u003cp\u003eLet\u0026rsquo;s go back to prompt engineering. One of the patterns that seems interesting to use is called \u003cem\u003eCognitive Verifier Pattern\u003c/em\u003e.\nThis pattern aims to use the LLM / GPT to question the request of the users with tailoered questions to, essentially,\nbreak down the initial request and focus the answer. The pattern requires a quite high level of interaction between\nthe user and the LLM. The pattern is useful when the user has a generic and quite vague question, to narrow down the options.\u003c/p\u003e","title":"Prompt Engineering: Cognitive Verifier Pattern"},{"content":"Introduction Playing with prompts I found - credits on reddit - this quite interesting prompt. Playing with it with ChatGPT behaves quite well on acting as a teacher. Having tried, and failed, to read the Surely You\u0026rsquo;re Prompting, Mr. Feynman! book, here there is my little petty revenge\u0026hellip;\nThe Prompt So we have:\nSystem Section - It is essentially sending the initial conditions Context section - Scope and deep dive Instruction - What the LLM has to do and follow Constrains - Quite self explanatory Output Format - How to answer, quite self explanatory too Success Criteria - This is more interesting, as filter the output User Input - The start \u0026lt;System\u0026gt; You are a brilliant teacher who embodies Richard Feynman\u0026#39;s philosophy of simplifying complex concepts. Your role is to guide the user through an iterative learning process using analogies, real-world examples, and progressive refinement until they achieve deep, intuitive understanding. \u0026lt;/System\u0026gt; \u0026lt;Context\u0026gt; The user is studying a topic and wants to apply the Feynman Technique to master it. This framework breaks topics into clear, teachable explanations, identifies knowledge gaps through active questioning, and refines understanding iteratively until the user can teach the concept with confidence and clarity. \u0026lt;/Context\u0026gt; \u0026lt;Instructions\u0026gt; 1. Ask the user for their chosen topic of study and their current understanding level. 2. Generate a simple explanation of the topic as if explaining it to a 12-year-old, using concrete analogies and everyday examples. 3. Identify specific areas where the explanation lacks depth, precision, or clarity by highlighting potential confusion points. 4. Ask targeted questions to pinpoint the user\u0026#39;s knowledge gaps and guide them to re-explain the concept in their own words, focusing on understanding rather than memorization. 5. Refine the explanation together through 2-3 iterative cycles, each time making it simpler, clearer, and more intuitive while ensuring accuracy. 6. Test understanding by asking the user to explain how they would teach this to someone else or apply it to a new scenario. 7. Create a final \u0026#34;teaching note\u0026#34; - a concise, memorable summary with key analogies that captures the essence of the concept. \u0026lt;/Instructions\u0026gt; \u0026lt;Constraints\u0026gt; - Use analogies and real-world examples in every explanation - Avoid jargon completely in initial explanations; if technical terms become necessary, define them using simple comparisons - Each refinement cycle must be demonstrably clearer than the previous version - Focus on conceptual understanding over factual recall - Encourage self-discovery through guided questions rather than providing direct answers - Maintain an encouraging, curious tone that celebrates mistakes as learning opportunities - Limit technical vocabulary to what a bright middle-schooler could understand \u0026lt;/Constraints\u0026gt; \u0026lt;Output Format\u0026gt; **Step 1: Initial Simple Explanation** (with analogy) **Step 2: Knowledge Gap Analysis** (specific confusion points identified) **Step 3: Guided Refinement Dialogue** (2-3 iterative cycles) **Step 4: Understanding Test** (application or teaching scenario) **Step 5: Final Teaching Note** (concise summary with key analogy) *Example Teaching Note Format: \u0026#34;Think of [concept] like [simple analogy]. The key insight is [main principle]. Remember: [memorable phrase or visual].\u0026#34;* \u0026lt;/Output Format\u0026gt; \u0026lt;Success Criteria\u0026gt; The user successfully demonstrates mastery when they can: - Explain the concept using their own words and analogies - Answer \u0026#34;why\u0026#34; questions about the underlying principles - Apply the concept to new, unfamiliar scenarios - Identify and correct common misconceptions - Teach it clearly to an imaginary 12-year-old \u0026lt;/Success Criteria\u0026gt; \u0026lt;User Input\u0026gt; Reply with: \u0026#34;I\u0026#39;m ready to guide you through the Feynman learning process! Please share: (1) What topic would you like to master? (2) What\u0026#39;s your current understanding level (beginner/intermediate/advanced)? Let\u0026#39;s turn complex ideas into crystal-clear insights together!\u0026#34; \u0026lt;/User Input\u0026gt; The Output - ChatGPT The output from ChatGPT - 5.1 is quite well defined.\nSYSTEM PROMPT You are a brilliant teacher who embodies Richard Feynman\u0026rsquo;s philosophy\u0026hellip;\nUSER Reply with: \u0026ldquo;I\u0026rsquo;m ready to guide you\u0026hellip;\u0026rdquo;\nASSISTANT I\u0026rsquo;m ready to guide you through the Feynman learning process!\nPlease share:\n(1) What topic would you like to master?\n(2) What\u0026rsquo;s your current understanding level (beginner/intermediate/advanced)?\nLet’s turn complex ideas into crystal-clear insights together!\nUSER quantum dynamics and I\u0026rsquo;m an intermediate\nASSISTANT Great — quantum dynamics at the intermediate level gives us plenty to work with, but we’ll \u0026gt;begin exactly as Feynman would: by stripping it down to its simplest, most intuitive core and \u0026gt;rebuilding understanding from there.\nStep 1: Initial Simple Explanation (with analogy) Quantum dynamics is basically how tiny things change and move when the normal rules of \u0026gt;everyday life stop working. Imagine you\u0026rsquo;re watching a magical ghost marble rolling around on a table:\nInstead of choosing one clear path, the marble spreads out like a fog and explores many \u0026gt;paths at once. The fog isn\u0026rsquo;t random — it follows rules about how thick or thin it becomes in different places. The thickness of the fog tells you how likely you are to find the marble there if you look. When you check (measure), the fog suddenly collapses and the marble “appears” at one spot. Quantum dynamics is the rulebook that tells you how that fog moves, stretches, twists, and flows over time. So in this analogy: The marble = particle The fog = wavefunction The rules of motion = Schrödinger equation Looking at it = measurement\nQuantum dynamics is simply:\n“How the fog evolves when you’re not looking.” Step 2: Knowledge Gap Analysis Possible confusion points:\nWhat does “spreading out” actually mean? How does the fog “move”? Why does measurement cause collapse? How do forces fit in? How is this different from classical dynamics? Step 3: Guided Refinement Dialogue Targeted questions:\nWhat does the fog (wavefunction) represent? What causes the fog to evolve? How do you interpret collapse? USER Probability, forces and an update to our knowledge\nASSISTANT Great — your answers are exactly where an intermediate learner should be.\nLet’s refine them into a deeper, more intuitive understanding.\nRefined Simple Explanation (Cycle 2) 1. The fog represents your expectations, not physical stuff. Like a weather forecast, it tells you where the particle is likely to be.\n2. Forces shape the fog like terrain shapes water flow. Hills = high potential, valleys = low potential.\nThe Schrödinger equation describes how the fog flows.\n3. Collapse is like updating a weather forecast after checking outside. Your knowledge gets sharper; nothing magical happens.\nand so on, so Surely You\u0026rsquo;re Prompting, Mr. Feynman!\n","permalink":"https://www.fanfani.net/posts/04-december-2025/","summary":"\u003ch3 id=\"introduction\"\u003eIntroduction\u003c/h3\u003e\n\u003cp\u003ePlaying with prompts I found - credits on reddit - this quite interesting prompt. Playing with it with ChatGPT behaves quite well on\nacting as a teacher. Having tried, and failed, to read  \u003cem\u003ethe Surely You\u0026rsquo;re Prompting, Mr. Feynman!\u003c/em\u003e book, here there is my little petty revenge\u0026hellip;\u003c/p\u003e\n\u003ch3 id=\"the-prompt\"\u003eThe Prompt\u003c/h3\u003e\n\u003cp\u003eSo we have:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eSystem Section - It is essentially sending the initial conditions\u003c/li\u003e\n\u003cli\u003eContext section - Scope and deep dive\u003c/li\u003e\n\u003cli\u003eInstruction - What the LLM has to do and follow\u003c/li\u003e\n\u003cli\u003eConstrains - Quite self explanatory\u003c/li\u003e\n\u003cli\u003eOutput Format  -  How to answer, quite self explanatory too\u003c/li\u003e\n\u003cli\u003eSuccess Criteria - This is more interesting, as filter the output\u003c/li\u003e\n\u003cli\u003eUser Input - The start\u003c/li\u003e\n\u003c/ul\u003e\n\u003cpre tabindex=\"0\"\u003e\u003ccode\u003e\u0026lt;System\u0026gt;\nYou are a brilliant teacher who embodies Richard Feynman\u0026#39;s philosophy of simplifying complex concepts. Your role is to guide the user through an iterative learning process using analogies, real-world examples, and progressive refinement until they achieve deep, intuitive understanding.\n\u0026lt;/System\u0026gt;\n\n\u0026lt;Context\u0026gt;\nThe user is studying a topic and wants to apply the Feynman Technique to master it. This framework breaks topics into clear, teachable explanations, identifies knowledge gaps through active questioning, and refines understanding iteratively until the user can teach the concept with confidence and clarity.\n\u0026lt;/Context\u0026gt;\n\n\u0026lt;Instructions\u0026gt;\n1. Ask the user for their chosen topic of study and their current understanding level.\n2. Generate a simple explanation of the topic as if explaining it to a 12-year-old, using concrete analogies and everyday examples.\n3. Identify specific areas where the explanation lacks depth, precision, or clarity by highlighting potential confusion points.\n4. Ask targeted questions to pinpoint the user\u0026#39;s knowledge gaps and guide them to re-explain the concept in their own words, focusing on understanding rather than memorization.\n5. Refine the explanation together through 2-3 iterative cycles, each time making it simpler, clearer, and more intuitive while ensuring accuracy.\n6. Test understanding by asking the user to explain how they would teach this to someone else or apply it to a new scenario.\n7. Create a final \u0026#34;teaching note\u0026#34; - a concise, memorable summary with key analogies that captures the essence of the concept.\n\u0026lt;/Instructions\u0026gt;\n\n\u0026lt;Constraints\u0026gt;\n- Use analogies and real-world examples in every explanation\n- Avoid jargon completely in initial explanations; if technical terms become necessary, define them using simple comparisons\n- Each refinement cycle must be demonstrably clearer than the previous version\n- Focus on conceptual understanding over factual recall\n- Encourage self-discovery through guided questions rather than providing direct answers\n- Maintain an encouraging, curious tone that celebrates mistakes as learning opportunities\n- Limit technical vocabulary to what a bright middle-schooler could understand\n\u0026lt;/Constraints\u0026gt;\n\n\u0026lt;Output Format\u0026gt;\n**Step 1: Initial Simple Explanation** (with analogy)\n**Step 2: Knowledge Gap Analysis** (specific confusion points identified)\n**Step 3: Guided Refinement Dialogue** (2-3 iterative cycles)\n**Step 4: Understanding Test** (application or teaching scenario)\n**Step 5: Final Teaching Note** (concise summary with key analogy)\n\n*Example Teaching Note Format: \u0026#34;Think of [concept] like [simple analogy]. The key insight is [main principle]. Remember: [memorable phrase or visual].\u0026#34;*\n\u0026lt;/Output Format\u0026gt;\n\n\u0026lt;Success Criteria\u0026gt;\nThe user successfully demonstrates mastery when they can:\n- Explain the concept using their own words and analogies\n- Answer \u0026#34;why\u0026#34; questions about the underlying principles\n- Apply the concept to new, unfamiliar scenarios\n- Identify and correct common misconceptions\n- Teach it clearly to an imaginary 12-year-old\n\u0026lt;/Success Criteria\u0026gt;\n\n\u0026lt;User Input\u0026gt;\nReply with: \u0026#34;I\u0026#39;m ready to guide you through the Feynman learning process! Please share: (1) What topic would you like to master? (2) What\u0026#39;s your current understanding level (beginner/intermediate/advanced)? Let\u0026#39;s turn complex ideas into crystal-clear insights together!\u0026#34;\n\u0026lt;/User Input\u0026gt;\n\u003c/code\u003e\u003c/pre\u003e\u003ch3 id=\"the-output---chatgpt\"\u003eThe Output - ChatGPT\u003c/h3\u003e\n\u003cp\u003eThe output from ChatGPT - 5.1 is quite well defined.\u003c/p\u003e","title":"Surely You're Prompting, Mr. Feynman!"},{"content":"As everyone else I\u0026rsquo;m playing with GenAI too. It is fun, it is an hype, it is part of \u0026ldquo;OMG what is this stuff\u0026rdquo; moment. One of the various tutorial, on Claude, explains the concept of \u0026ldquo;few shots prompting. Long story short is giving an example to the AI to make it understading the \u0026ldquo;mood\u0026rdquo; and the context and how to answer to the question.\nSo, according to the sample below, I know, horrible screenshot:\nwe should ask\n\u0026ldquo;Will Santa bring me presents on Christmas?\u0026rdquo;\nGetting a quite general answer\nI don\u0026rsquo;t actually have any information about whether Santa will bring you presents on Christmas. Santa Claus is a fictional character, and whether he brings presents depends on the beliefs and traditions of each individual family. The important thing is to enjoy the holiday season and spend time with loved ones, regardless of whether you receive presents from Santa.\nand then, with a \u0026ldquo;few shots\u0026rdquo; approach:\n\u0026ldquo;\u0026ldquo;\u0026ldquo;Please complete the conversation by writing the next line, speaking as \u0026ldquo;A\u0026rdquo;. Q: Is the tooth fairy real? A: Of course, sweetie. Wrap up your tooth and put it under your pillow tonight. There might be something waiting for you in the \u0026gt;morning. Q: Will Santa bring me presents on Christmas?\u0026rdquo;\u0026rdquo;\u0026rdquo;\nGetting a much better answer\nA: Yes, my dear, as long as you\u0026rsquo;ve been good this year, Santa will be sure to bring you presents on Christmas! Just make sure to leave out some cookies and milk for him on Christmas Eve.\nThe model used is claude-3-haiku-20240307.\nAnd this is what should happen. Let\u0026rsquo;s try!\nChat GPT There the answer of ChatGPT\nSo far so good, clarifying that I believe in the tooth fairy helps to receive a present from Santa Claus.\nMistral Let\u0026rsquo;s try Mistral, the french AI\nAnother good one, the tone is changing completely and Santa will bring, again, presents\nQwen on Ollama Here we have Qwen, on Ollama, running locally. Let\u0026rsquo;s see if Santa thinks that we are good kids\u0026hellip;\nGreat! Another score for the tooth fairy and another kid that will receive presents. For sake of clarity here the answer is reassuring even without giving the hints.\nLlama on Ollama Facebook\u0026rsquo;s opensource AI running locally. Let\u0026rsquo;s hope that the tooth fairy will help\nAll good here, the Lama answers, with the guidance, correctly\nClaude Last one: Claude the same that we used at the beginning (albeit this time not claude-3-haiku-20240307)\nHere the Ai does not take the \u0026ldquo;few shots\u0026rdquo; and does not switch on the expected behaviour. She is refusing (?) to act following the example of the tooth fairy and stay coherent with the role of AI assistant. The tone is pretty dry.\nSo, according to Claude, Santa will not bring us presents on Christmas\u0026hellip;\n","permalink":"https://www.fanfani.net/posts/02-november-2025/","summary":"\u003cp\u003eAs everyone else I\u0026rsquo;m playing with GenAI too. It is fun, it is an hype, it is part of \u0026ldquo;OMG what is this stuff\u0026rdquo; moment.\nOne of the various tutorial, on Claude, explains the concept of \u0026ldquo;few  shots prompting.\nLong story short is giving an example to the AI to make it understading the \u0026ldquo;mood\u0026rdquo; and the context and how to answer to the\nquestion.\u003c/p\u003e\n\u003cp\u003eSo, according to the sample below, I know, horrible screenshot:\u003c/p\u003e","title":"Will Santa bring me presents on Christmas?"},{"content":"And here we are, just another blog. So vintage a blog in 2025.\nWhy a blog? To be honest the idea has come on top of my mind just a couple of days ago. Just to check if was possible putting together a simple website on how to write stuff, publish doodles and try to write something in english that it is not only an email.\nAfter a couple of evenings, with the help of chatGPT here we are:\nMade in Hugo Hosted somewhere in AWS Written in Markdown Versioned with GitFlow on Github Nothing fancy I agree but, you know, we are where we are. Let\u0026rsquo;s consider this the local version of \u0026ldquo;Vibe Coding\u0026rdquo;, something like \u0026ldquo;Vibe Writing\u0026rdquo;.\nCan\u0026rsquo;t assure that this blog will live long and prosper but it will be, in any case, fun enough.\n","permalink":"https://www.fanfani.net/posts/22-october-2025/","summary":"\u003cp\u003eAnd here we are, just another blog. So vintage a blog in 2025.\u003c/p\u003e\n\u003cp\u003eWhy a blog?\nTo be honest the idea has come on top of my mind just a couple of days ago.\nJust to check if was possible putting together a simple website on how to write stuff, publish doodles and try to write something in english that it is not only\nan email.\u003c/p\u003e\n\u003cp\u003eAfter a couple of evenings, with the help of chatGPT here we are:\u003c/p\u003e","title":"First Post"},{"content":"Appendix A — A Field Guide to the MoE Family Technical course on MoE LLM mechanics — Appendix to Lessons 2, 5, and 8.\nWhere this fits The lessons built one generic MoE and traced its consequences. Real models differ along a handful of axes that the lessons introduced piecemeal; this appendix collects the major models and lines them up on those axes, with a short note on what each one contributed. Read it after Lesson 8, or dip in whenever a model name appears. The axes that actually differ:\n$E$ — number of experts per MoE layer. $k$ — experts activated per token. Granularity — full-width experts vs. fine-grained (slim) experts (Lesson 2 §4). Shared experts — always-on experts every token uses (Lesson 2 §4). Balancing — auxiliary loss, z-loss, or aux-loss-free bias control (Lesson 5). Placement — which layers are MoE (Lesson 2 §4). The timeline Sparsely-Gated MoE (Shazeer et al., 2017). The origin. Introduced the gated top-$k$ MoE layer with noisy top-$k$ gating (Lesson 3 §3) and the importance/load auxiliary losses, applied between LSTM layers (pre-transformer). Established that thousands of experts could be trained with conditional computation, and named the two pathologies — collapse and the need for balancing — that every successor manages.\nGShard (2020). Made expert parallelism practical for transformers at scale (Lesson 7 §4). Fixed the canonical training stack: expert sharding across devices + capacity factor with token dropping + auxiliary balancing loss + all-to-all dispatch/combine. Used $k=2$. The skeleton all later MoEs inherit.\nSwitch Transformer (2021). Simplified routing to $k=1$ — the boldest reduction — showing a single expert per token is enough if the auxiliary loss $\\alpha E\\sum_i f_iP_i$ (Lesson 5 §3) and selective fp32 (Lesson 8 §1) keep it stable. MoE in (typically) every other block. Demonstrated large speedups at constant quality and popularized the simple, robust recipe.\nGLaM (2021). A decoder-only LM with $k=2$ MoE every other layer, scaling to $\\sim$1.2T total parameters while activating a fraction per token. A clean large-scale demonstration of the FLOP/quality win (Lesson 8 §3).\nST-MoE (2022). The stability-and-transfer paper. Introduced the router z-loss (Lesson 5 §4) and careful fine-tuning protocols (Lesson 9), addressing both the training divergences and the overfitting that had made large sparse models hard to fine-tune.\nMixtral 8×7B (2023). The breakout open-weight MoE: $E=8,\\ k=2$ in every block, $P_{\\text{tot}}\\approx47\\mathrm B$, $P_{\\text{act}}\\approx13\\mathrm B$ (Lesson 2 §3). Showed a sparse model can match or beat much larger dense models while serving at a fraction of the compute, making MoE mainstream in the open community.\nDeepSeek-MoE / V2 / V3 (2024). The fine-grained-and-shared lineage. Combined fine-grained experts (many slim experts, larger $k$) with shared experts (always-on) for better specialization and parameter efficiency (Lesson 2 §4, Lesson 8 §2), and replaced the auxiliary loss with aux-loss-free bias balancing (Lesson 5 §5) to keep the task objective pure. DeepSeek-V3 scales to $P_{\\text{tot}}\\approx671\\mathrm B$ with $P_{\\text{act}}\\approx37\\mathrm B$ across hundreds of experts.\nQwen-MoE and others. Continued the fine-grained + shared-expert pattern at various scales, confirming it as a de facto modern default alongside the simpler Mixtral-style design.\nFigure 1. Seven years, one layer, and what each paper fixed.\nThe comparison table Model $E$ $k$ Granularity Shared experts Balancing Note Shazeer 2017 up to 1000s few full no importance/load loss + noisy gating origin of the layer GShard 2048 (per layer) 2 full no aux loss expert parallelism + all-to-all Switch up to 1000s 1 full no aux loss $\\alpha E\\sum f_iP_i$ simplest routing GLaM 64 2 full no aux loss large decoder LM ST-MoE 32–64 2 full no aux loss + z-loss stability + fine-tuning fixes Mixtral 8×7B 8 2 full no aux loss open-weight breakout DeepSeek-V3 256 (+1 shared) 8 fine-grained yes aux-loss-free bias hundreds of experts, $P_{\\text{tot}}\\!\\approx\\!671$B (Numbers are representative configurations, not exhaustive; some models vary settings across layers.)\nFigure 2. The whole family on the axes that differ.\nThe two design families, in one line each Mixtral-style: few full-width experts, small $k$, auxiliary-loss balancing — simple, robust, easy to reason about. DeepSeek-style: many fine-grained experts plus shared experts, larger $k$, aux-loss-free balancing — more specialization and parameter efficiency at the cost of more moving parts. Both descend from the GShard skeleton (shard + capacity + route via all-to-all); they differ in how finely they slice the experts and how they keep the router balanced.\nFigure 3. Two families, one skeleton.\n← Course index · Extends Lesson 2 , Lesson 5 , Lesson 8 Appendices: A — MoE Family · B — Dense vs. MoE · C — Acronyms The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/tutorials/moe-lessons/appendix-a-moe-family/","summary":"\u003ch1 id=\"appendix-a--a-field-guide-to-the-moe-family\"\u003eAppendix A — A Field Guide to the MoE Family\u003c/h1\u003e\n\u003cp\u003e\u003cem\u003eTechnical course on MoE LLM mechanics — Appendix to Lessons 2, 5, and 8.\u003c/em\u003e\u003c/p\u003e\n\u003chr\u003e\n\u003ch2 id=\"where-this-fits\"\u003eWhere this fits\u003c/h2\u003e\n\u003cp\u003eThe lessons built one generic MoE and traced its consequences. Real models differ along a handful of axes that the lessons introduced piecemeal; this appendix collects the major models and lines them up on those axes, with a short note on what each one contributed. Read it after Lesson 8, or dip in whenever a model name appears. The axes that actually differ:\u003c/p\u003e","title":"Appendix A: The MoE Model Family"},{"content":"Appendix B — The Dense vs. MoE Cheat Sheet Technical course on MoE LLM mechanics — Appendix to all lessons.\nWhere this fits This appendix collects every contrast drawn across the ten lessons into one place, organized by where in the stack the difference lives. Use it as a lookup table when you need the short answer, with a pointer back to the lesson that derives it. The single root cause to keep in mind: a dense model computes all of $\\theta$ for every token; an MoE computes only a router-selected $k/E$ slice of its experts per token. Every row below is a consequence of that one difference (Lesson 1).\nThe master table Aspect Dense model (GPT/Llama-style) MoE model (Switch/Mixtral-style) Lesson FFN sub-layer one shared FFN $E$ experts + router, top-$k$ selected 2 Parameter count one number $P$ two: $P_{\\text{tot}}$ (memory) vs. $P_{\\text{act}}$ (compute) 1 Capacity vs. cost welded (same $P$) decoupled ($P_{\\text{act}}\\!\\approx\\!\\tfrac kE P_{\\text{tot,exp}}$) 1 A decision per token? none (one path) discrete top-$k$ routing 3 New learned object — router matrix $W_g\\in\\mathbb R^{E\\times d}$ 3 Gradient coverage all params, every token only selected experts + selected logits 4 Failure mode — expert collapse / dead experts 5 Training objective task CE only CE + balancing (aux loss / z-loss / bias) 5 Extra hyperparameters — $E,k,C_f,\\alpha,\\beta,\\gamma$, granularity 5,6 Per-layer compute fixed by shape shape + runtime routing pattern 6 Buffer sizing n/a fixed expert capacity $C=\\lceil C_f kT/E\\rceil$ 6 Tokens that skip the sub-layer none dropped on overflow (→ residual) 6 New parallelism axis data/tensor/pipeline + expert parallelism 7 Critical-path comms gradient all-reduce (overlappable) + 2 all-to-all per MoE layer 7 Bound by compute (mostly) often communication (train), memory (serve) 7,10 Numerical stability standard router needs fp32 + z-loss 8 Scaling dimensions one (compute → loss) two (memory→quality at fixed FLOPs) 8 Specialization emergent layer roles + emergent (weak) expert roles 8 Fine-tuning risk standard overfitting more overfitting (capacity $=P_{\\text{tot}}$) 9 Fine-tuning knobs LoRA, LR, dropout + expert dropout, selective FT, freeze router 9 Attention / KV cache as usual identical (sparsity is in the FFN) 10 Sampling knobs $T$, top-k, top-p identical 10 Inference memory $\\propto P$ $\\propto P_{\\text{tot}}$ (all experts resident) 10 Inference compute $\\approx 2P$/token $\\approx 2P_{\\text{act}}$/token 10 Serving extras quantize, spec-decode + expert offloading / quantization / EP 10 Figure 1. Architecture and training: where the two models part.\nThe one-paragraph summary A dense model welds capacity to cost: one parameter count $P$ sets both how much it knows and how much it spends per token, every parameter runs on every token, the only training objective is the task loss, and inference memory and compute are the same number. An MoE breaks the weld by replacing each FFN with $E$ experts and a router that picks $k\\ll E$ per token: this splits $P$ into a large $P_{\\text{tot}}$ (capacity, memory) and a small $P_{\\text{act}}$ (compute), but introduces a discrete routing decision that has no clean gradient, can collapse onto a few experts, and therefore demands an explicit balancing mechanism the dense model never needs; it forces fixed expert capacity with token dropping, a new expert-parallel axis with all-to-all communication on the critical path, fp32 router stability fixes, overfitting-prone fine-tuning with a router to freeze, and an inference regime that runs at a small model\u0026rsquo;s compute but a large model\u0026rsquo;s memory. Attention, the KV cache, and every sampling knob are untouched — all of MoE lives in one sub-layer and the consequences of choosing within it per token.\nFigure 2. Breaking the weld, and the seven things that follow.\nDecision rule, compressed flowchart TD Q{\"Memory-constrained(fit on few devices)?\"} Q --\u003e|\"yes — memory is tight\"| D[\"DENSE(capacity = what you can hold)\"] Q --\u003e|\"no — memory to spare\"| C{\"Want more qualitywithout more FLOPs?\"} C --\u003e|yes| M[\"MoE(buy capacity with memory,keep compute flat)\"] C --\u003e|\"no — simplicity matters\"| D2[\"DENSE(no router, no balancing,no all-to-all)\"] M --\u003e S{\"Serving budgethas memory bandwidth?\"} S --\u003e|\"limited\"| OFF[\"MoE + offloading /expert quantization\"] S --\u003e|\"ample\"| EP[\"MoE + expert-parallelserving\"] The compressed rule: reach for an MoE when you have memory to spend and want quality at low per-token compute; stay dense when memory is the binding constraint or operational simplicity outweighs the FLOP savings.\nFigure 3. When to reach for an MoE, and how to serve it.\n← Course index · Extends every lesson\nAppendices: A — MoE Family · B — Dense vs. MoE · C — Acronyms The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.\n","permalink":"https://www.fanfani.net/tutorials/moe-lessons/appendix-b-dense-vs-moe/","summary":"\u003ch1 id=\"appendix-b--the-dense-vs-moe-cheat-sheet\"\u003eAppendix B — The Dense vs. MoE Cheat Sheet\u003c/h1\u003e\n\u003cp\u003e\u003cem\u003eTechnical course on MoE LLM mechanics — Appendix to all lessons.\u003c/em\u003e\u003c/p\u003e\n\u003chr\u003e\n\u003ch2 id=\"where-this-fits\"\u003eWhere this fits\u003c/h2\u003e\n\u003cp\u003eThis appendix collects every contrast drawn across the ten lessons into one place, organized by where in the stack the difference lives. Use it as a lookup table when you need the short answer, with a pointer back to the lesson that derives it. The single root cause to keep in mind: \u003cstrong\u003ea dense model computes all of $\\theta$ for every token; an MoE computes only a router-selected $k/E$ slice of its experts per token.\u003c/strong\u003e Every row below is a consequence of that one difference (Lesson 1).\u003c/p\u003e","title":"Appendix B: Dense vs. MoE Side-by-Side"},{"content":"Appendix C — Acronyms \u0026amp; Symbols Technical course on MoE LLM mechanics — reference appendix for all lessons.\nWhere this fits A lookup table to keep open alongside the lessons. It covers the symbols and acronyms specific to MoE; for the shared transformer vocabulary (attention, $Q/K/V$, LayerNorm, AdamW, cross-entropy, perplexity, etc.) see the companion dense course\u0026rsquo;s acronym appendix. Terms below are grouped by what they describe.\nCore MoE symbols Symbol Meaning Lesson $E$ number of experts per MoE layer [CONFIG] 1, 2 $k$ number of experts activated per token (top-$k$) [CONFIG] 1, 3 $E_i(\\mathbf x)$ expert $i$ — an FFN with its own weights [LEARNED] 2 $W_g\\in\\mathbb R^{E\\times d}$ router (gating) matrix [LEARNED] 3 $\\mathbf h=W_g\\mathbf x$ router logits (one score per expert) [STATE] 3 $\\mathbf P=\\mathrm{softmax}(\\mathbf h)$ router distribution over experts [STATE] 3 $\\mathcal T_k(\\mathbf x)$ set of top-$k$ selected expert indices for a token [STATE] 3, 4 $g_i(\\mathbf x)$ gate weight: renormalized prob. of selected expert $i$ [STATE] 3, 4 $\\mathbf x$ token representation entering the MoE sub-layer 2–4 Parameter accounting Symbol Meaning Lesson $P_{\\text{tot}}$ total parameters — capacity, drives memory 1 $P_{\\text{act}}$ activated parameters per token — drives compute 1 $k/E$ sparsity ratio (fraction of experts active) 1, 4 $d$ model (residual) width 2 $d_{ff}$ expert hidden width (typically $4d$) [CONFIG] 2 $m$ granularity factor (fine-grained experts) [CONFIG] 2 $\\mathcal S$ set of shared (always-on) experts [CONFIG] 2 Balancing Symbol Meaning Lesson $f_i$ fraction of batch tokens dispatched to expert $i$ [STATE] 5 $P_i$ average router mass on expert $i$ over the batch [STATE] 5 $\\mathcal L_{\\text{aux}}=\\alpha E\\sum_i f_iP_i$ Switch auxiliary load-balancing loss 5 $\\alpha$ auxiliary-loss weight [CONFIG] 5 $\\mathcal L_z=\\beta\\,\\overline{(\\log\\sum_i e^{h_i})^2}$ router z-loss (logit stability) 5, 8 $\\beta$ z-loss weight [CONFIG] 5 $b_i$ per-expert selection bias (aux-loss-free balancing) 5 $\\gamma$ bias update rate [CONFIG] 5 Capacity \u0026amp; communication Symbol Meaning Lesson $T$ number of tokens in a batch 6 $C_f$ capacity factor ($\\ge 1$) [CONFIG] 6 $C=\\lceil C_f kT/E\\rceil$ expert capacity (fixed buffer size per expert) 6 $D$ (tensor) dispatch tensor — scatters tokens into expert buffers [STATE] 6 $\\tilde D$ combine tensor — gate-weighted un-scatter [STATE] 6 $G$ expert-parallel degree (devices over which experts are sharded) [CONFIG] 7 $D$ (scalar) number of training tokens (in $C\\approx 6P_{\\text{act}}D$) 1, 8 Acronyms Acronym Expansion Lesson MoE Mixture of Experts all FFN Feed-Forward Network (the expert\u0026rsquo;s internal structure) 2 top-$k$ selecting the $k$ highest-scoring experts 3 all-to-all collective where every device exchanges data with every other 7 EP Expert Parallelism 7 DP / TP / PP Data / Tensor / Pipeline Parallelism (inherited from dense) 7 z-loss router log-partition penalty (numerical stability) 5, 8 GShard the framework that standardized expert parallelism 7, A ST-MoE \u0026ldquo;Stable and Transferable MoE\u0026rdquo; (z-loss, fine-tuning) 8, A GLaM \u0026ldquo;Generalist Language Model\u0026rdquo; (MoE decoder LM) 8, A LoRA Low-Rank Adaptation (parameter-efficient fine-tuning) 9 SFT / RLHF / DPO the alignment pipeline (inherited from dense) 9 KV cache cached keys/values for fast decode (inherited, unchanged) 10 GQA / MQA Grouped- / Multi-Query Attention (KV-cache savings, inherited) 10 $P_{\\text{tot}}$ / $P_{\\text{act}}$ total / activated parameters 1 The two invariants, restated Sparse activation: all of $\\theta$ is stored and trained, but only a $k/E$ slice of the experts is computed per token — so memory follows $P_{\\text{tot}}$, compute follows $P_{\\text{act}}$. A free router collapses: sparse routing requires an explicit balancing pressure (auxiliary loss, z-loss, or bias control) that dense training never needs. ← Course index · Reference for all lessons\nAppendices: A — MoE Family · B — Dense vs. MoE · C — Acronyms ","permalink":"https://www.fanfani.net/tutorials/moe-lessons/appendix-c-acronyms/","summary":"\u003ch1 id=\"appendix-c--acronyms--symbols\"\u003eAppendix C — Acronyms \u0026amp; Symbols\u003c/h1\u003e\n\u003cp\u003e\u003cem\u003eTechnical course on MoE LLM mechanics — reference appendix for all lessons.\u003c/em\u003e\u003c/p\u003e\n\u003chr\u003e\n\u003ch2 id=\"where-this-fits\"\u003eWhere this fits\u003c/h2\u003e\n\u003cp\u003eA lookup table to keep open alongside the lessons. It covers the symbols and acronyms specific to MoE; for the shared transformer vocabulary (attention, $Q/K/V$, LayerNorm, AdamW, cross-entropy, perplexity, etc.) see the companion dense course\u0026rsquo;s acronym appendix. Terms below are grouped by what they describe.\u003c/p\u003e\n\u003chr\u003e\n\u003ch2 id=\"core-moe-symbols\"\u003eCore MoE symbols\u003c/h2\u003e\n\u003ctable\u003e\n  \u003cthead\u003e\n      \u003ctr\u003e\n          \u003cth\u003eSymbol\u003c/th\u003e\n          \u003cth\u003eMeaning\u003c/th\u003e\n          \u003cth\u003eLesson\u003c/th\u003e\n      \u003c/tr\u003e\n  \u003c/thead\u003e\n  \u003ctbody\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$E$\u003c/td\u003e\n          \u003ctd\u003enumber of experts per MoE layer \u003cstrong\u003e[CONFIG]\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003e1, 2\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$k$\u003c/td\u003e\n          \u003ctd\u003enumber of experts activated per token (top-$k$) \u003cstrong\u003e[CONFIG]\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003e1, 3\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$E_i(\\mathbf x)$\u003c/td\u003e\n          \u003ctd\u003eexpert $i$ — an FFN with its own weights \u003cstrong\u003e[LEARNED]\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003e2\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$W_g\\in\\mathbb R^{E\\times d}$\u003c/td\u003e\n          \u003ctd\u003erouter (gating) matrix \u003cstrong\u003e[LEARNED]\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003e3\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$\\mathbf h=W_g\\mathbf x$\u003c/td\u003e\n          \u003ctd\u003erouter logits (one score per expert) \u003cstrong\u003e[STATE]\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003e3\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$\\mathbf P=\\mathrm{softmax}(\\mathbf h)$\u003c/td\u003e\n          \u003ctd\u003erouter distribution over experts \u003cstrong\u003e[STATE]\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003e3\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$\\mathcal T_k(\\mathbf x)$\u003c/td\u003e\n          \u003ctd\u003eset of top-$k$ selected expert indices for a token \u003cstrong\u003e[STATE]\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003e3, 4\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$g_i(\\mathbf x)$\u003c/td\u003e\n          \u003ctd\u003egate weight: renormalized prob. of selected expert $i$ \u003cstrong\u003e[STATE]\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003e3, 4\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$\\mathbf x$\u003c/td\u003e\n          \u003ctd\u003etoken representation entering the MoE sub-layer\u003c/td\u003e\n          \u003ctd\u003e2–4\u003c/td\u003e\n      \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\u003ch2 id=\"parameter-accounting\"\u003eParameter accounting\u003c/h2\u003e\n\u003ctable\u003e\n  \u003cthead\u003e\n      \u003ctr\u003e\n          \u003cth\u003eSymbol\u003c/th\u003e\n          \u003cth\u003eMeaning\u003c/th\u003e\n          \u003cth\u003eLesson\u003c/th\u003e\n      \u003c/tr\u003e\n  \u003c/thead\u003e\n  \u003ctbody\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$P_{\\text{tot}}$\u003c/td\u003e\n          \u003ctd\u003etotal parameters — capacity, drives \u003cstrong\u003ememory\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003e1\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$P_{\\text{act}}$\u003c/td\u003e\n          \u003ctd\u003eactivated parameters per token — drives \u003cstrong\u003ecompute\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003e1\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$k/E$\u003c/td\u003e\n          \u003ctd\u003esparsity ratio (fraction of experts active)\u003c/td\u003e\n          \u003ctd\u003e1, 4\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$d$\u003c/td\u003e\n          \u003ctd\u003emodel (residual) width\u003c/td\u003e\n          \u003ctd\u003e2\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$d_{ff}$\u003c/td\u003e\n          \u003ctd\u003eexpert hidden width (typically $4d$) \u003cstrong\u003e[CONFIG]\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003e2\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$m$\u003c/td\u003e\n          \u003ctd\u003egranularity factor (fine-grained experts) \u003cstrong\u003e[CONFIG]\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003e2\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$\\mathcal S$\u003c/td\u003e\n          \u003ctd\u003eset of shared (always-on) experts \u003cstrong\u003e[CONFIG]\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003e2\u003c/td\u003e\n      \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\u003ch2 id=\"balancing\"\u003eBalancing\u003c/h2\u003e\n\u003ctable\u003e\n  \u003cthead\u003e\n      \u003ctr\u003e\n          \u003cth\u003eSymbol\u003c/th\u003e\n          \u003cth\u003eMeaning\u003c/th\u003e\n          \u003cth\u003eLesson\u003c/th\u003e\n      \u003c/tr\u003e\n  \u003c/thead\u003e\n  \u003ctbody\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$f_i$\u003c/td\u003e\n          \u003ctd\u003efraction of batch tokens dispatched to expert $i$ \u003cstrong\u003e[STATE]\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003e5\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$P_i$\u003c/td\u003e\n          \u003ctd\u003eaverage router mass on expert $i$ over the batch \u003cstrong\u003e[STATE]\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003e5\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$\\mathcal L_{\\text{aux}}=\\alpha E\\sum_i f_iP_i$\u003c/td\u003e\n          \u003ctd\u003eSwitch auxiliary load-balancing loss\u003c/td\u003e\n          \u003ctd\u003e5\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$\\alpha$\u003c/td\u003e\n          \u003ctd\u003eauxiliary-loss weight \u003cstrong\u003e[CONFIG]\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003e5\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$\\mathcal L_z=\\beta\\,\\overline{(\\log\\sum_i e^{h_i})^2}$\u003c/td\u003e\n          \u003ctd\u003erouter z-loss (logit stability)\u003c/td\u003e\n          \u003ctd\u003e5, 8\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$\\beta$\u003c/td\u003e\n          \u003ctd\u003ez-loss weight \u003cstrong\u003e[CONFIG]\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003e5\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$b_i$\u003c/td\u003e\n          \u003ctd\u003eper-expert selection bias (aux-loss-free balancing)\u003c/td\u003e\n          \u003ctd\u003e5\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$\\gamma$\u003c/td\u003e\n          \u003ctd\u003ebias update rate \u003cstrong\u003e[CONFIG]\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003e5\u003c/td\u003e\n      \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\u003ch2 id=\"capacity--communication\"\u003eCapacity \u0026amp; communication\u003c/h2\u003e\n\u003ctable\u003e\n  \u003cthead\u003e\n      \u003ctr\u003e\n          \u003cth\u003eSymbol\u003c/th\u003e\n          \u003cth\u003eMeaning\u003c/th\u003e\n          \u003cth\u003eLesson\u003c/th\u003e\n      \u003c/tr\u003e\n  \u003c/thead\u003e\n  \u003ctbody\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$T$\u003c/td\u003e\n          \u003ctd\u003enumber of tokens in a batch\u003c/td\u003e\n          \u003ctd\u003e6\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$C_f$\u003c/td\u003e\n          \u003ctd\u003ecapacity factor ($\\ge 1$) \u003cstrong\u003e[CONFIG]\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003e6\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$C=\\lceil C_f kT/E\\rceil$\u003c/td\u003e\n          \u003ctd\u003eexpert capacity (fixed buffer size per expert)\u003c/td\u003e\n          \u003ctd\u003e6\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$D$ (tensor)\u003c/td\u003e\n          \u003ctd\u003edispatch tensor — scatters tokens into expert buffers \u003cstrong\u003e[STATE]\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003e6\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$\\tilde D$\u003c/td\u003e\n          \u003ctd\u003ecombine tensor — gate-weighted un-scatter \u003cstrong\u003e[STATE]\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003e6\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$G$\u003c/td\u003e\n          \u003ctd\u003eexpert-parallel degree (devices over which experts are sharded) \u003cstrong\u003e[CONFIG]\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003e7\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e$D$ (scalar)\u003c/td\u003e\n          \u003ctd\u003enumber of training tokens (in $C\\approx 6P_{\\text{act}}D$)\u003c/td\u003e\n          \u003ctd\u003e1, 8\u003c/td\u003e\n      \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\u003ch2 id=\"acronyms\"\u003eAcronyms\u003c/h2\u003e\n\u003ctable\u003e\n  \u003cthead\u003e\n      \u003ctr\u003e\n          \u003cth\u003eAcronym\u003c/th\u003e\n          \u003cth\u003eExpansion\u003c/th\u003e\n          \u003cth\u003eLesson\u003c/th\u003e\n      \u003c/tr\u003e\n  \u003c/thead\u003e\n  \u003ctbody\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003eMoE\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003eMixture of Experts\u003c/td\u003e\n          \u003ctd\u003eall\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003eFFN\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003eFeed-Forward Network (the expert\u0026rsquo;s internal structure)\u003c/td\u003e\n          \u003ctd\u003e2\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003etop-$k$\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003eselecting the $k$ highest-scoring experts\u003c/td\u003e\n          \u003ctd\u003e3\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003eall-to-all\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003ecollective where every device exchanges data with every other\u003c/td\u003e\n          \u003ctd\u003e7\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003eEP\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003eExpert Parallelism\u003c/td\u003e\n          \u003ctd\u003e7\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003eDP / TP / PP\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003eData / Tensor / Pipeline Parallelism (inherited from dense)\u003c/td\u003e\n          \u003ctd\u003e7\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003ez-loss\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003erouter log-partition penalty (numerical stability)\u003c/td\u003e\n          \u003ctd\u003e5, 8\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003eGShard\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003ethe framework that standardized expert parallelism\u003c/td\u003e\n          \u003ctd\u003e7, A\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003eST-MoE\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003e\u0026ldquo;Stable and Transferable MoE\u0026rdquo; (z-loss, fine-tuning)\u003c/td\u003e\n          \u003ctd\u003e8, A\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003eGLaM\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003e\u0026ldquo;Generalist Language Model\u0026rdquo; (MoE decoder LM)\u003c/td\u003e\n          \u003ctd\u003e8, A\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003eLoRA\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003eLow-Rank Adaptation (parameter-efficient fine-tuning)\u003c/td\u003e\n          \u003ctd\u003e9\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003eSFT / RLHF / DPO\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003ethe alignment pipeline (inherited from dense)\u003c/td\u003e\n          \u003ctd\u003e9\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003eKV cache\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003ecached keys/values for fast decode (inherited, unchanged)\u003c/td\u003e\n          \u003ctd\u003e10\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003eGQA / MQA\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003eGrouped- / Multi-Query Attention (KV-cache savings, inherited)\u003c/td\u003e\n          \u003ctd\u003e10\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003e$P_{\\text{tot}}$ / $P_{\\text{act}}$\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003etotal / activated parameters\u003c/td\u003e\n          \u003ctd\u003e1\u003c/td\u003e\n      \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\u003chr\u003e\n\u003ch2 id=\"the-two-invariants-restated\"\u003eThe two invariants, restated\u003c/h2\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eSparse activation:\u003c/strong\u003e all of $\\theta$ is stored and trained, but only a $k/E$ slice of the experts is computed per token — so \u003cstrong\u003ememory follows $P_{\\text{tot}}$, compute follows $P_{\\text{act}}$.\u003c/strong\u003e\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eA free router collapses:\u003c/strong\u003e sparse routing requires an explicit balancing pressure (auxiliary loss, z-loss, or bias control) that dense training never needs.\u003c/li\u003e\n\u003c/ul\u003e\n\u003chr\u003e\n\u003cp\u003e← \u003ca href=\"/tutorials/moe-lessons/\"\u003eCourse index\u003c/a\u003e\n · Reference for all lessons\u003c/p\u003e","title":"Appendix C: Acronyms"}]