How LLMs Work on Ship of Theseus

How LLMs Work on Ship of Theseushttps://www.fanfani.net/tutorials/llm-lessons/Recent content in How LLMs Work on Ship of TheseusHugo -- 0.152.2en-usSun, 23 Aug 2026 00:00:00 +0000The Big Picturehttps://www.fanfani.net/tutorials/llm-lessons/lesson-01-the-big-picture/Mon, 01 Jan 0001 00:00:00 +0000https://www.fanfani.net/tutorials/llm-lessons/lesson-01-the-big-picture/<p>Lesson 1 — The Big Picture: Training vs. Inference A 10-lesson technical course on the mechanics of large language models. Audience: technically comfortable, new to ML. Prerequisites: linear algebra (matrix products, vector spaces), multivariate calculus (gradients, the chain rule), and basic probability (distributions, expectation). We use standard notation throughout and define it as we go. Course table of contents # Lesson Core technical content 1 The Big Picture (this file) The map $f(x;\theta)$; training vs. inference; parameters vs. config vs. optimizer state; FLOP and memory accounting 2 Tokenization & Embeddings BPE; the embedding map $E\in\mathbb R^{V\times d}$; one-hot lookups; sinusoidal vs. learned vs. rotary (RoPE) positions 3 Self-Attention $\mathrm{softmax}(QK^\top/\sqrt{d_k})V$; the $\sqrt{d_k}$ variance argument; softmax Jacobian; causal masking; multi-head; $O(n^2d)$ cost 4 The Transformer Block FFN and parameter budget; GELU/SwiGLU; residual gradient flow; LayerNorm forward + backward; RMSNorm; pre- vs. post-norm 5 Output Head & Objective Unembedding $W_U$; weight tying; the autoregressive factorization; the $\partial\mathcal L/\partial \mathbf z=\hat{\mathbf p}-\mathbf y$ identity 6 The Loss Function Cross-entropy as $H(p)+D_{\mathrm{KL}}(p\|\hat p)$; nats vs. bits; perplexity; logit-gradient derivation 7 Backprop & Gradient Descent Reverse-mode autodiff; VJPs; per-layer adjoint equations; activation checkpointing; vanishing/exploding gradients 8 Optimizers, LR & Batches SGD→momentum→Adam with bias-correction derivation; AdamW; clipping; mixed precision; scaling laws 9 Fine-Tuning, SFT & RLHF SFT loss masking; LoRA $W+\frac{\alpha}{r}BA$; Bradley–Terry reward; PPO clipped objective; DPO loss 10 Inference in Depth Greedy/beam/sampling; temperature, top-k, top-p; KV-cache memory; GQA/MQA; speculative decoding A Appendix A — Acronyms Every acronym and symbol used in the course, expanded and explained in plain language B Appendix B — Formulas All key equations, collected by lesson with a one-line reminder of what each says Two invariants run through every lesson and are worth stating before anything else:</p><p><a href="https://www.fanfani.net/tutorials/llm-lessons/lesson-01-the-big-picture/">Read the full post →</a></p>Tokenization and Embeddingshttps://www.fanfani.net/tutorials/llm-lessons/lesson-02-tokenization-and-embeddings/Mon, 01 Jan 0001 00:00:00 +0000https://www.fanfani.net/tutorials/llm-lessons/lesson-02-tokenization-and-embeddings/<p>Lesson 2 — Tokenization & Embeddings Technical course on LLM mechanics — Lesson 2 of 10. Recap of Lesson 1 A model is a function $f(x;\theta)$ from a token sequence to a categorical distribution over the next token, and the chain rule $P_\theta(x_{1:n})=\prod_t P_\theta(x_t\mid x_{\lt t})$ makes that sufficient to model whole sequences. Training minimizes $\mathcal L(\theta)$ via a forward/backward/update loop and is the only phase that writes $\theta$; inference evaluates $f$ at frozen $\theta$. We separate three kinds of numbers — [LEARNED] parameters, [CONFIG] hyperparameters (which fix the shape of $\theta$), and [STATE] transient values — and quantified cost as $\approx 6PD$ FLOPs to train, $\approx 2P$/token to infer.</p><p><a href="https://www.fanfani.net/tutorials/llm-lessons/lesson-02-tokenization-and-embeddings/">Read the full post →</a></p>Self-Attentionhttps://www.fanfani.net/tutorials/llm-lessons/lesson-03-self-attention/Mon, 01 Jan 0001 00:00:00 +0000https://www.fanfani.net/tutorials/llm-lessons/lesson-03-self-attention/<p>Lesson 3 — Self-Attention Technical course on LLM mechanics — Lesson 3 of 10. Recap of Lesson 2 The bottom of $f$ maps a string to $H^{(0)}\in\mathbb R^{n\times d}$ via a fixed tokenizer $\tau$ ([CONFIG], BPE-learned offline) and a learned embedding $E\in\mathbb R^{V\times d}$ ([LEARNED], $Vd$ parameters, row-sparse gradient). Because attention is permutation-equivariant, position is injected separately — sinusoidal, learned, or RoPE (relative position inside the dot product). We now derive the attention operator that turns these independent per-token vectors into contextualized ones, and analyze its cost.</p><p><a href="https://www.fanfani.net/tutorials/llm-lessons/lesson-03-self-attention/">Read the full post →</a></p>The Transformer Blockhttps://www.fanfani.net/tutorials/llm-lessons/lesson-04-the-transformer-block/Mon, 01 Jan 0001 00:00:00 +0000https://www.fanfani.net/tutorials/llm-lessons/lesson-04-the-transformer-block/<p>Lesson 4 — The Transformer Block Technical course on LLM mechanics — Lesson 4 of 10. Recap of Lesson 3 Self-attention computes $A V$ with $A=\mathrm{softmax}(QK^\top/\sqrt{d_k}+M)$, a row-stochastic matrix of learned asymmetric similarities; the $\sqrt{d_k}$ divisor keeps softmax off its zero-gradient saturation (Jacobian $\mathrm{diag}(\mathbf a)-\mathbf a\mathbf a^\top$); the causal mask enforces autoregression. Multi-head attention uses $h$ subspaces ($d_k=d/h$), costs $\approx4d^2$ params/layer and $O(n^2d)$ compute. The attention matrix is [STATE]. We now assemble the full block and analyze depth, normalization, and the parameter budget.</p><p><a href="https://www.fanfani.net/tutorials/llm-lessons/lesson-04-the-transformer-block/">Read the full post →</a></p>Output Head and Objectivehttps://www.fanfani.net/tutorials/llm-lessons/lesson-05-output-head-and-objective/Mon, 01 Jan 0001 00:00:00 +0000https://www.fanfani.net/tutorials/llm-lessons/lesson-05-output-head-and-objective/<p>Lesson 5 — The Output Head & the Language-Modeling Objective Technical course on LLM mechanics — Lesson 5 of 10. Recap of Lesson 4 A block is two pre-norm residual sub-layers: multi-head attention and a position-wise MLP ($W_2\phi(W_1\mathbf x+\mathbf b_1)+\mathbf b_2$, $d_{ff}=4d$), the latter supplying nonlinear capacity and $\approx\tfrac23$ of the $P\approx12Nd^2+2Vd$ parameters. Residuals give the layer Jacobian $I+\partial\mathcal F$ (identity gradient path → trainable depth); LayerNorm/RMSNorm standardize features with a learned gain. Stacking $N$ blocks yields $H^{(N)}\in\mathbb R^{n\times d}$. We now map $H^{(N)}$ to next-token probabilities and state the training objective precisely.</p><p><a href="https://www.fanfani.net/tutorials/llm-lessons/lesson-05-output-head-and-objective/">Read the full post →</a></p>The Loss Functionhttps://www.fanfani.net/tutorials/llm-lessons/lesson-06-the-loss-function/Mon, 01 Jan 0001 00:00:00 +0000https://www.fanfani.net/tutorials/llm-lessons/lesson-06-the-loss-function/<p>Lesson 6 — The Loss Function Technical course on LLM mechanics — Lesson 6 of 10. Recap of Lesson 5 The head produces logits $\mathbf z=\mathbf h^\top W_U$ and softmax gives $\hat{\mathbf p}\in\Delta^{V-1}$. The objective is maximum likelihood of observed tokens = minimum average cross-entropy $\mathcal L=-\frac1n\sum_t\log\hat p_{t,x_t}$, self-supervised because the label at $t$ is $x_{t+1}$. The softmax+cross-entropy pairing yields the clean gradient $\partial\mathcal L/\partial\mathbf z=\hat{\mathbf p}-\mathbf y$. This lesson develops the loss information-theoretically, derives its key properties, and connects it to perplexity.</p><p><a href="https://www.fanfani.net/tutorials/llm-lessons/lesson-06-the-loss-function/">Read the full post →</a></p>Backpropagation and Gradient Descenthttps://www.fanfani.net/tutorials/llm-lessons/lesson-07-backpropagation-and-gradient-descent/Mon, 01 Jan 0001 00:00:00 +0000https://www.fanfani.net/tutorials/llm-lessons/lesson-07-backpropagation-and-gradient-descent/<p>Lesson 7 — Backpropagation & Gradient Descent Technical course on LLM mechanics — Lesson 7 of 10. Recap of Lesson 6 Cross-entropy $=H(p)+D_{\mathrm{KL}}(p\|\hat p)$, so minimizing it minimizes KL to the true distribution, floored by language’s entropy $H(p)$. With one-hot targets it is $-\log\hat p_{x_t}$, a proper scoring rule (hence calibration, hence why we don’t train on accuracy). Perplexity $e^{\mathcal L}$ is the effective branching factor. The logit gradient is $\partial\mathcal L/\partial\mathbf z=\hat{\mathbf p}-\mathbf y$. This lesson turns that single gradient into $\nabla_\theta\mathcal L\in\mathbb R^P$ and specifies the update — the precise mechanics of what gets written.</p><p><a href="https://www.fanfani.net/tutorials/llm-lessons/lesson-07-backpropagation-and-gradient-descent/">Read the full post →</a></p>Optimizers, Learning Rate and Batcheshttps://www.fanfani.net/tutorials/llm-lessons/lesson-08-optimizers-learning-rate-batches/Mon, 01 Jan 0001 00:00:00 +0000https://www.fanfani.net/tutorials/llm-lessons/lesson-08-optimizers-learning-rate-batches/<p>Lesson 8 — Optimizers, Learning Rate & Batches Technical course on LLM mechanics — Lesson 8 of 10. Recap of Lesson 7 Backprop computes $\nabla_\theta\mathcal L\in\mathbb R^P$ in one reverse sweep via VJPs ($\bar W=\bar{\mathbf u}\mathbf x^\top$, $\bar{\mathbf x}=W^\top\bar{\mathbf u}$), seeded by $\hat{\mathbf p}-\mathbf y$. The bare update is $\theta\leftarrow\theta-\eta\nabla_\theta\mathcal L$. We estimate the gradient on mini-batches (unbiased, variance $\propto1/|B|$). Residuals/normalization/scaling keep the Jacobian product conditioned; checkpointing tames memory. This lesson specifies the optimizer that maps $g\mapsto$ update, the schedule on $\eta$, and the regularizers — all [CONFIG], plus the optimizer’s [STATE].</p><p><a href="https://www.fanfani.net/tutorials/llm-lessons/lesson-08-optimizers-learning-rate-batches/">Read the full post →</a></p>Fine-Tuning, SFT and RLHFhttps://www.fanfani.net/tutorials/llm-lessons/lesson-09-finetuning-sft-rlhf/Mon, 01 Jan 0001 00:00:00 +0000https://www.fanfani.net/tutorials/llm-lessons/lesson-09-finetuning-sft-rlhf/<p>Lesson 9 — Fine-Tuning, SFT & RLHF Technical course on LLM mechanics — Lesson 9 of 10. Recap of Lesson 8 Adam/AdamW maps the mini-batch gradient to a per-coordinate adaptive step via two EMAs ($\mathbf m,\mathbf v$), bias-corrected and divided as $\hat{\mathbf m}/(\sqrt{\hat{\mathbf v}}+\epsilon)$, with decoupled weight decay; its $2P$ buffers dominate training memory. A warmup→cosine-decay $\eta$ schedule with gradient clipping, large batches, and data-scale regularization, planned via scaling laws, completes pretraining. The product is a base model: an excellent estimator of $P_\theta(x_t\mid x_{\lt t})$ over web text — but not an aligned assistant. This lesson covers the post-training stages and, per the course theme, exactly what each updates. The throughline: every stage is the same gradient descent of Lessons 6–8; only the data and the loss’s source change.</p><p><a href="https://www.fanfani.net/tutorials/llm-lessons/lesson-09-finetuning-sft-rlhf/">Read the full post →</a></p>Inference in Depthhttps://www.fanfani.net/tutorials/llm-lessons/lesson-10-inference-in-depth/Mon, 01 Jan 0001 00:00:00 +0000https://www.fanfani.net/tutorials/llm-lessons/lesson-10-inference-in-depth/<p>Lesson 10 — Inference in Depth Technical course on LLM mechanics — Lesson 10 of 10. Recap of Lesson 9 Post-training aligns the base model via the same gradient descent: SFT (response-masked next-token CE on demonstrations; LoRA for efficiency), then RLHF — a Bradley–Terry reward model $r_\phi$ followed by KL-anchored PPO, or the single-loss DPO — each updating only the policy or reward model, leaving the mechanism unchanged. The output is one frozen, aligned $\theta$. We now return to inference, the regime where $\theta$ is fixed and only the forward pass runs, and specify every decoding knob and the systems machinery that makes generation fast. Per the course theme: nothing here updates a parameter; every knob is [CONFIG] and every buffer is [STATE].</p><p><a href="https://www.fanfani.net/tutorials/llm-lessons/lesson-10-inference-in-depth/">Read the full post →</a></p>Appendix A — Acronymshttps://www.fanfani.net/tutorials/llm-lessons/appendix-a-acronyms/Mon, 01 Jan 0001 00:00:00 +0000https://www.fanfani.net/tutorials/llm-lessons/appendix-a-acronyms/<p>Appendix A — Acronyms & Abbreviations A glossary for the 10-lesson course. Each entry gives the expansion, a plain-language explanation, and where in the course it first matters. Grouped by theme so related terms sit together; use your viewer’s find (Ctrl/Cmd-F) to jump to a specific term. This is a lookup table, not a chapter — keep it open beside the lessons and jump in when an unfamiliar acronym appears. The groups below trace the same arc the course does: the model’s components (Lesson 1–5), then the machinery that trains and aligns it (Lessons 6–9), then how it is run (Lesson 10). The symbol table at the end collects the single letters that recur as standard notation throughout.</p><p><a href="https://www.fanfani.net/tutorials/llm-lessons/appendix-a-acronyms/">Read the full post →</a></p>Appendix A — Chinchilla Scaling Lawshttps://www.fanfani.net/tutorials/llm-lessons/appendix-a-chinchilla-scaling-laws/Mon, 01 Jan 0001 00:00:00 +0000https://www.fanfani.net/tutorials/llm-lessons/appendix-a-chinchilla-scaling-laws/<p>Appendix A — The Chinchilla Law: Compute-Optimal Scaling Technical course on LLM mechanics — Appendix to Lesson 8. Where this fits This is a side-quest off Lesson 8. There, scaling laws appeared as the planning layer of a training run — the rule that tells you, before you spend a dollar of compute, how big to build and how much to feed. This appendix follows that thread to its practical conclusion: the Chinchilla recipe. Lesson 8 §7 introduced scaling laws this way: pretraining loss falls as a smooth power law in parameters and data,</p><p><a href="https://www.fanfani.net/tutorials/llm-lessons/appendix-a-chinchilla-scaling-laws/">Read the full post →</a></p>Appendix B — Formulashttps://www.fanfani.net/tutorials/llm-lessons/appendix-b-formulas/Mon, 01 Jan 0001 00:00:00 +0000https://www.fanfani.net/tutorials/llm-lessons/appendix-b-formulas/<p>Appendix B — Mathematical Formulas Every key equation from the course, collected in one place and grouped by lesson. Each formula has a one-line reminder of what it says; follow the lesson link for the full derivation. Read top to bottom, the equations retrace the whole arc: the model is defined as a function (Lesson 1), built up layer by layer (Lessons 2–5), scored against a target (Lessons 6), and improved by gradient descent and its refinements (Lessons 7–9) before being run to generate text (Lesson 10). Used as a reference, it is a quick way to recover a formula’s meaning and find where it was derived. Each symbol here matches its lesson exactly; nothing is restated in a simpler form.</p><p><a href="https://www.fanfani.net/tutorials/llm-lessons/appendix-b-formulas/">Read the full post →</a></p>Appendix C — Attention Variantshttps://www.fanfani.net/tutorials/llm-lessons/appendix-c-attention-variants/Mon, 01 Jan 0001 00:00:00 +0000https://www.fanfani.net/tutorials/llm-lessons/appendix-c-attention-variants/<p>Appendix C — A Field Guide to Attention Variants Technical course on LLM mechanics — Appendix to Lessons 3 and 10. Where this fits This is a side-quest off Lessons 3 and 10. Lesson 3 built the attention mechanism; Lesson 10 showed why, at inference time, the obvious version is too expensive to serve. The names that get thrown around — MQA, GQA, sliding-window, RoPE, FlashAttention — are all answers to one of those two pressures (what the task needs, or what the hardware can afford), and this appendix sorts them out. Lesson 3 defined the one operation everything here is a variant of — scaled dot-product attention,</p><p><a href="https://www.fanfani.net/tutorials/llm-lessons/appendix-c-attention-variants/">Read the full post →</a></p>Appendix D — Positional Encodingshttps://www.fanfani.net/tutorials/llm-lessons/appendix-d-positional-encodings/Mon, 01 Jan 0001 00:00:00 +0000https://www.fanfani.net/tutorials/llm-lessons/appendix-d-positional-encodings/<p>Appendix D — Positional Encodings Technical course on LLM mechanics — Appendix to Lesson 2 (§4, “Injecting position”). A catalogue of the ways a transformer is told where each token sits. This is a side-quest off Lesson 2 §4: that section introduced three schemes in passing because position is one of the few model components that is genuinely a design choice rather than a forced consequence of the math, and the choice has quietly become one of the most active areas of LLM engineering. This appendix collects every family that matters in practice, with the math, the property that defines each, and where it is [CONFIG] (chosen, no parameters) versus [LEARNED] (trained).</p><p><a href="https://www.fanfani.net/tutorials/llm-lessons/appendix-d-positional-encodings/">Read the full post →</a></p>