How Diffusion Models Work on Ship of Theseus

The Big Picture

Mon, 01 Jan 0001 00:00:00 +0000

Lesson 1 — The Big Picture: What Is a Diffusion Model? Course map. Ten lessons. We start from intuition, build the forward (noising) process, derive the reverse (denoising) process from the variational lower bound, collapse it into the famous one-line training loss, study the network that implements it, write the training loop, then the sampling loop, then the fast/deterministic samplers and the score-based view, and finally conditioning, guidance and latent diffusion. Each lesson opens with a recap of the previous one and closes with a conclusion.

Read the full post →

The Forward Process

Mon, 01 Jan 0001 00:00:00 +0000

Lesson 2 — The Forward Process: Destroying Structure with Noise Recap of Lesson 1. A diffusion model is built on an asymmetry: a fixed forward process $q$ that gradually turns data $x_0$ into pure noise $x_T$, and a learned reverse process $p_\theta$ that denoises step by step. Generation = running the learned reverse from noise. We met the single forward step $q(x_t\mid x_{t-1}) = \mathcal{N}(x_t; \sqrt{1-\beta_t}\,x_{t-1}, \beta_t I)$ and the assumed-Gaussian reverse step. Now we make the forward side completely rigorous.

Read the full post →

The Reverse Process

Mon, 01 Jan 0001 00:00:00 +0000

Lesson 3 — The Reverse Process: Learning to Denoise Recap of Lesson 2. The forward process is a fixed Gaussian Markov chain, $q(x_t\mid x_{t-1}) = \mathcal{N}(x_t; \sqrt{1-\beta_t}\,x_{t-1}, \beta_t I)$. With $\alpha_t = 1-\beta_t$ and $\bar\alpha_t=\prod_{s\le t}\alpha_s$, we proved the closed-form jump $x_t = \sqrt{\bar\alpha_t}\,x_0 + \sqrt{1-\bar\alpha_t}\,\epsilon$, so any noise level is one operation away, and $x_T\approx\mathcal N(0,I)$. The noise $\epsilon$ is the natural training target. Now we build the reverse direction.

Read the full post →

The Variational Lower Bound

Mon, 01 Jan 0001 00:00:00 +0000

Lesson 4 — The Variational Lower Bound (ELBO) Recap of Lesson 3. Because forward steps are tiny, the reverse step is Gaussian; we model it as $p_\theta(x_{t-1}\mid x_t)=\mathcal N(x_{t-1};\mu_\theta(x_t,t),\Sigma_\theta)$. The true reverse $q(x_{t-1}\mid x_t)$ is intractable, but conditioning on $x_0$ yields a tractable Gaussian posterior $q(x_{t-1}\mid x_t,x_0)=\mathcal N(\tilde\mu_t,\tilde\beta_t I)$ with closed-form $\tilde\mu_t$ and $\tilde\beta_t$. Rewriting $\tilde\mu_t$ in terms of the noise $\epsilon$ showed that predicting $\epsilon$ recovers the ideal denoiser. This lesson explains why matching that posterior is the principled objective.

Read the full post →

The Simplified Objective

Mon, 01 Jan 0001 00:00:00 +0000

Lesson 5 — The Simplified Training Objective Recap of Lesson 4. We maximize a variational lower bound $L = L_T + \sum_{t\ge2}L_{t-1} + L_0$. The prior term $L_T$ is a parameter-free constant; the reconstruction term $L_0$ is minor; the denoising terms $L_{t-1}$ carry the learning. With fixed reverse variance $\sigma_t^2 I$, each reduces to a weighted squared error between the network’s reverse mean $\mu_\theta(x_t,t)$ and the analytic posterior mean $\tilde\mu_t(x_t,x_0)$: $L_{t-1}=\frac{1}{2\sigma_t^2}\|\tilde\mu_t-\mu_\theta\|^2 + C$. Now we collapse this into one clean loss.

Read the full post →

U-Net Architecture

Mon, 01 Jan 0001 00:00:00 +0000

Lesson 6 — The Network: U-Net and Time Conditioning Recap of Lesson 5. The $\epsilon$-parameterization $\mu_\theta = \frac{1}{\sqrt{\alpha_t}}(x_t - \frac{\beta_t}{\sqrt{1-\bar\alpha_t}}\epsilon_\theta)$ makes schedule factors cancel, collapsing the variational objective into the simple loss $L_{\text{simple}} = \mathbb E_{t,x_0,\epsilon}\,\|\epsilon - \epsilon_\theta(\sqrt{\bar\alpha_t}x_0+\sqrt{1-\bar\alpha_t}\epsilon,\,t)\|^2$. The network is a noise predictor with unlimited, cheaply-generated training pairs. This lesson opens the box: what is $\epsilon_\theta$ actually made of? We have spent five lessons describing a job without once meeting the worker who does it: take a noisy image, hand back the noise hidden inside it, and do so at every level of corruption with the same pair of hands. This lesson opens the box we have only ever named by its task. Inside is not a featureless slab but a particular, well-argued shape — a network that narrows until it can see the whole picture at a glance, widens again to put the grain back, and ferries fine detail across the gap on dedicated wires — and that is told, every time, exactly how noisy the image before it is.

Read the full post →

Training Algorithm

Mon, 01 Jan 0001 00:00:00 +0000

Lesson 7 — The Training Algorithm in Practice Recap of Lesson 6. The noise predictor $\epsilon_\theta$ is an image-to-image U-Net — contracting encoder, attention bottleneck, expanding decoder, skip connections — with the timestep injected via a sinusoidal embedding passed through an MLP and added (as scale/shift) inside every residual block. Modern systems may use a Diffusion Transformer instead, but the loss is unchanged. This lesson assembles the loss and the network into the full training phase.

Read the full post →

DDPM Sampling

Mon, 01 Jan 0001 00:00:00 +0000

Lesson 8 — The Sampling Algorithm: Inference with DDPM Recap of Lesson 7. Training is a five-line loop: sample $x_0$, sample $t\sim\mathcal U\{1,T\}$, sample $\epsilon\sim\mathcal N(0,I)$, form $x_t=\sqrt{\bar\alpha_t}x_0+\sqrt{1-\bar\alpha_t}\epsilon$ in one shot, and minimize $\|\epsilon-\epsilon_\theta(x_t,t)\|^2$. Crucially, training never runs the reverse chain; it uses the variational surrogate. EMA weights and a good schedule make it stable. Now we finally run the reverse chain to generate. Until now the reverse process has been a thing we reasoned about but never ran. This lesson finally turns the crank. We begin with a square of pure static — no image, nothing, just a draw from a Gaussian — and put the same patient question to the trained network a thousand times over: what is the noise here, and what lies one notch beneath it? Slowly, out of the static, a face that never existed sharpens into view. The price of that thousand-fold patience is the very problem the next lesson exists to solve.

Read the full post →

DDIM and Score-Based Models

Mon, 01 Jan 0001 00:00:00 +0000

Lesson 9 — Faster & Deterministic Sampling (DDIM) and the Score View Recap of Lesson 8. Inference runs the reverse chain: start at $x_T\sim\mathcal N(0,I)$ and iterate $x_{t-1}=\frac{1}{\sqrt{\alpha_t}}(x_t-\frac{1-\alpha_t}{\sqrt{1-\bar\alpha_t}}\epsilon_\theta) + \sigma_t z$ down to $x_0$. The deterministic part denoises; $\sigma_t z$ keeps samples diverse. An equivalent “predict $\hat x_0$, re-noise to $t{-}1$” view clarifies each step. The cost — ~$T=1000$ sequential network calls — is diffusion’s main weakness. We now make sampling fast and connect to score-based generative modeling.

Read the full post →

Conditioning and Guidance

Mon, 01 Jan 0001 00:00:00 +0000

Lesson 10 — Conditioning, Guidance, and the Modern Landscape Recap of Lesson 9. DDIM rewrites the reverse step as “predict $\hat x_0$ + direction term + optional $\sigma_t z$”; with $\sigma_t=0$ it is deterministic, lets us skip timesteps (20–50 ≈ 1000), and gives an invertible latent space. The score view shows $\epsilon_\theta \approx -\sqrt{1-\bar\alpha_t}\,\nabla_x\log q_t(x)$, recasting generation as solving a reverse-time SDE or its probability-flow ODE (fast ODE solvers → ~10–20 steps). The $v$-parameterization keeps targets well-scaled. This final lesson adds steering and scale.

Read the full post →

Appendix A — Acronyms

Mon, 01 Jan 0001 00:00:00 +0000

Appendix A — Acronyms & Abbreviations How to read this list. Every acronym and abbreviation that appears anywhere in the ten lessons is collected here, expanded in full, and explained in one or two sentences of plain language. The right-hand column points you to the lesson(s) where the term does real work, so you can jump back to its first proper use. Terms are grouped by theme rather than alphabetically, because related ideas are easier to learn together; an alphabetical quick-index follows at the end.

Read the full post →

Appendix B — Formulas

Mon, 01 Jan 0001 00:00:00 +0000

Appendix B — Mathematical Formulas How to read this list. Every load-bearing equation from the ten lessons is gathered here in one place, grouped by topic and ordered roughly as they appear in the course. Each entry gives the formula and a one-line gloss of what it does and where to find its derivation. Math uses LaTeX delimiters ($...$, $$...$$) and renders in any LaTeX-aware Markdown viewer. A symbol glossary closes the appendix.

Read the full post →