Appendix B — Mathematical Formulas

How to read this list. Every load-bearing equation from the ten lessons is gathered here in one place, grouped by topic and ordered roughly as they appear in the course. Each entry gives the formula and a one-line gloss of what it does and where to find its derivation. Math uses LaTeX delimiters ($...$, $$...$$) and renders in any LaTeX-aware Markdown viewer. A symbol glossary closes the appendix.

Notation at a glance

$x_0$ is clean data; $x_1,\dots,x_T$ are increasingly noisy latents; $x_T\approx$ pure noise. The schedule is $\beta_t\in(0,1)$ with

$$ \alpha_t := 1-\beta_t, \qquad \bar\alpha_t := \prod_{s=1}^{t}\alpha_s . $$

$q$ is the fixed forward (noising) law, $p_\theta$ the learned reverse (denoising) law, $\epsilon\sim\mathcal N(0,I)$ standard noise, and $\epsilon_\theta$ the network’s noise predictor. Throughout, $a_t:=\sqrt{\bar\alpha_t}$ and $b_t:=\sqrt{1-\bar\alpha_t}$.

1 · The two processes (Lesson 1)

Reverse generative joint — the model factorizes as a Markov chain from noise to data:

$$ p_\theta(x_{0:T}) = p(x_T)\prod_{t=1}^{T} p_\theta(x_{t-1}\mid x_t), \qquad p(x_T)=\mathcal N(0,I). $$

Fixed forward chain — the frozen “encoder” that defines the latents:

$$ q(x_{1:T}\mid x_0)=\prod_{t=1}^T q(x_t\mid x_{t-1}), \qquad q(x_t\mid x_{t-1})=\mathcal N\!\big(\sqrt{1-\beta_t}\,x_{t-1},\,\beta_t I\big). $$

Assumed Gaussian reverse step — what the network must parameterize:

$$ p_\theta(x_{t-1}\mid x_t) = \mathcal N\!\left(x_{t-1};\ \mu_\theta(x_t,t),\ \Sigma_\theta(x_t,t)\right). $$

2 · The forward process (Lesson 2)

Single-step reparameterization:

$$ x_t = \sqrt{1-\beta_t}\, x_{t-1} + \sqrt{\beta_t}\,\epsilon_{t-1}, \qquad \epsilon_{t-1}\sim\mathcal{N}(0,I). $$

Closed-form jump to any noise level — the single most useful identity in diffusion; lets you reach step $t$ in one shot:

$$ \boxed{\,q(x_t\mid x_0) = \mathcal{N}\!\left(x_t;\ \sqrt{\bar\alpha_t}\,x_0,\ (1-\bar\alpha_t) I\right)\,} \quad\Longleftrightarrow\quad x_t = \sqrt{\bar\alpha_t}\,x_0 + \sqrt{1-\bar\alpha_t}\,\epsilon,\ \ \epsilon\sim\mathcal{N}(0,I). $$

Signal-to-noise ratio:

$$ \mathrm{SNR}(t) := \frac{\bar\alpha_t}{1-\bar\alpha_t}, \qquad \log\mathrm{SNR}(t) = \log\bar\alpha_t - \log(1-\bar\alpha_t). $$

Cosine schedule (Nichol & Dhariwal):

$$ \bar\alpha_t = \frac{f(t)}{f(0)}, \qquad f(t) = \cos\!\left(\frac{t/T + s}{1+s}\cdot\frac{\pi}{2}\right)^2 . $$

Continuous-time (VP-SDE) limit: $\ dx = -\tfrac12\beta(t)\,x\,dt + \sqrt{\beta(t)}\,dw\ $ with $\ \bar\alpha_t=\exp\!\big(-\!\int_0^t\beta(s)\,ds\big)$.

3 · The reverse process & tractable posterior (Lesson 3)

Bayes form of the posterior — tractable once we condition on $x_0$:

$$ q(x_{t-1}\mid x_t, x_0) = q(x_t\mid x_{t-1})\,\frac{q(x_{t-1}\mid x_0)}{q(x_t\mid x_0)} = \mathcal{N}\!\left(x_{t-1};\ \tilde\mu_t(x_t, x_0),\ \tilde\beta_t I\right). $$

Posterior variance:

$$ \boxed{\ \tilde\beta_t = \frac{1-\bar\alpha_{t-1}}{1-\bar\alpha_t}\,\beta_t\ } $$

Posterior mean (in $x_0,x_t$ form):

$$ \boxed{\ \tilde\mu_t(x_t,x_0) = \frac{\sqrt{\bar\alpha_{t-1}}\,\beta_t}{1-\bar\alpha_t}\,x_0 + \frac{\sqrt{\alpha_t}\,(1-\bar\alpha_{t-1})}{1-\bar\alpha_t}\,x_t\ } $$

Inverting the jump for $x_0$: $\ x_0 = \frac{1}{\sqrt{\bar\alpha_t}}\left(x_t - \sqrt{1-\bar\alpha_t}\,\epsilon\right)$.

Posterior mean (in noise form) — substitute the line above:

$$ \boxed{\ \tilde\mu_t(x_t, x_0) = \frac{1}{\sqrt{\alpha_t}}\left(x_t - \frac{\beta_t}{\sqrt{1-\bar\alpha_t}}\,\epsilon\right)\ } $$

4 · The variational lower bound (Lesson 4)

Jensen bound — turns an intractable log-integral into an expectation we can optimize:

$$ -\log p_\theta(x_0) = -\log \int q(x_{1:T}\mid x_0)\,\frac{p_\theta(x_{0:T})}{q(x_{1:T}\mid x_0)}\,dx_{1:T} \le \mathbb{E}_{q}\!\left[-\log\frac{p_\theta(x_{0:T})}{q(x_{1:T}\mid x_0)}\right] =: L. $$

Exact KL-gap identity — the bound’s slack is exactly a KL, hence $L\ge -\log p_\theta(x_0)$:

$$ \boxed{\ L - \big(-\log p_\theta(x_0)\big) = D_{\mathrm{KL}}\!\big(q(x_{1:T}\mid x_0)\,\|\,p_\theta(x_{1:T}\mid x_0)\big)\ \ge 0.\ } $$

ELBO decomposition into per-step terms:

$$ \boxed{, L = \underbrace{D_{\mathrm{KL}}!\big(q(x_T\mid x_0),|,p(x_T)\big)}_{L_T}

• \sum_{t=2}^{T}\underbrace{D_{\mathrm{KL}}!\big(q(x_{t-1}\mid x_t,x_0),|,p_\theta(x_{t-1}\mid x_t)\big)}{L{t-1}} \underbrace{-,\log p_\theta(x_0\mid x_1)}_{L_0} ,} $$

Gaussian KL (general, diagonal):

$$ D_{\mathrm{KL}} = \frac{1}{2}\!\left[\operatorname{tr}(\Sigma_2^{-1}\Sigma_1) + (\mu_2-\mu_1)^\top\Sigma_2^{-1}(\mu_2-\mu_1) - d + \log\frac{\det\Sigma_2}{\det\Sigma_1}\right]. $$

Denoising term under matched covariance — the KL collapses to a scaled mean gap:

$$ \boxed{\ L_{t-1} = \frac{1}{2\sigma_t^2}\,\big\|\tilde\mu_t(x_t,x_0) - \mu_\theta(x_t,t)\big\|^2 + C.\ } $$

5 · The simplified objective (Lesson 5)

Mean parameterization through predicted noise:

$$ \boxed{\ \mu_\theta(x_t, t) = \frac{1}{\sqrt{\alpha_t}}\left(x_t - \frac{\beta_t}{\sqrt{1-\bar\alpha_t}}\,\epsilon_\theta(x_t,t)\right)\ } $$

Per-timestep ELBO weight (what $L_{\text{simple}}$ drops by setting it to $1$):

$$ \boxed{\ w_t = \frac{\beta_t^2}{2\sigma_t^2\,\alpha_t(1-\bar\alpha_t)}\ } $$

Fully explicit ELBO denoising term:

$$ \boxed{\ L_{t-1} = \frac{\beta_t^2}{2\sigma_t^2\,\alpha_t(1-\bar\alpha_t)}\, \Big\|\,\epsilon - \epsilon_\theta\big(\sqrt{\bar\alpha_t}\,x_0 + \sqrt{1-\bar\alpha_t}\,\epsilon,\ t\big)\,\Big\|^2\ } $$

The simplified objective — the famous one-line loss:

$$ \boxed{\ L_{\text{simple}}(\theta) = \mathbb{E}_{t\sim\mathcal U\{1,T\},\ x_0\sim q,\ \epsilon\sim\mathcal N(0,I)}\Big[\big\|\,\epsilon - \epsilon_\theta(\sqrt{\bar\alpha_t}\,x_0 + \sqrt{1-\bar\alpha_t}\,\epsilon,\ t)\,\big\|^2\Big]\ } $$

v-parameterization — a balanced regression target:

$$ \boxed{\ v := \sqrt{\bar\alpha_t}\,\epsilon - \sqrt{1-\bar\alpha_t}\,x_0\ } $$

with inverse maps $\ \epsilon = \sqrt{1-\bar\alpha_t}\,x_t + \sqrt{\bar\alpha_t}\,v\ $ and $\ x_0 = \sqrt{\bar\alpha_t}\,x_t - \sqrt{1-\bar\alpha_t}\,v$.

Score connection: $\ \epsilon_\theta(x_t,t)\approx -\sqrt{1-\bar\alpha_t}\,\nabla_{x_t}\log q(x_t)$.

6 · U-Net & time conditioning (Lesson 6)

Sinusoidal time embedding:

$$ \text{emb}(t)_{2i} = \sin\!\left(\frac{t}{10000^{2i/d}}\right),\qquad \text{emb}(t)_{2i+1} = \cos\!\left(\frac{t}{10000^{2i/d}}\right),\qquad i=0,\dots,\tfrac{d}{2}-1. $$

Time MLP: $\ \tau = W_2\,\mathrm{SiLU}(W_1\,\text{emb}(t) + b_1) + b_2$.

FiLM / adaLN scale-shift injection:

$$ \boxed{\ (\gamma,\beta)=W_\tau\,\tau,\qquad h \leftarrow (1+\gamma)\odot \mathrm{Norm}(h) + \beta\ } $$

adaLN-Zero block (DiT): $\ h\leftarrow h+\alpha\odot\mathrm{Block}\big((1+\gamma)\,\mathrm{LN}(h)+\beta\big)$, with $\alpha$ initialized to $0$.

7 · The training algorithm (Lesson 7)

Algorithm 1 — Training. Repeat: sample $x_0\sim q(x_0)$, $t\sim\mathcal U\{1,\dots,T\}$, $\epsilon\sim\mathcal N(0,I)$; form $x_t = \sqrt{\bar\alpha_t}\,x_0 + \sqrt{1-\bar\alpha_t}\,\epsilon$; take a gradient step on $\nabla_\theta \|\epsilon - \epsilon_\theta(x_t, t)\|^2$.

Importance-sampling estimator for the full VLB (variance reduction):

$$ \hat L = \frac{1}{p(t)}\,L_{t-1},\qquad \mathbb{E}_{t\sim p}[\hat L] = \sum_t L_{t-1},\qquad p(t)\propto\sqrt{\mathbb{E}[L_{t-1}^2]}. $$

Min-SNR-$\gamma$ weight — rebalances easy vs. hard noise levels:

$$ w_t = \frac{\min\{\mathrm{SNR}(t),\,\gamma\}}{\mathrm{SNR}(t)},\qquad \mathrm{SNR}(t) = \frac{\bar\alpha_t}{1-\bar\alpha_t}. $$

8 · DDPM sampling (Lesson 8)

Ancestral reverse step — the inference workhorse:

$$ \boxed{\ x_{t-1} = \frac{1}{\sqrt{\alpha_t}}\left(x_t - \frac{1-\alpha_t}{\sqrt{1-\bar\alpha_t}}\,\epsilon_\theta(x_t, t)\right) + \sigma_t\, z, \qquad z\sim\mathcal N(0,I)\ } $$

Two variance choices: $\ \sigma_t^2 = \beta_t\ $ (upper bound) or $\ \sigma_t^2 = \tilde\beta_t = \frac{1-\bar\alpha_{t-1}}{1-\bar\alpha_t}\,\beta_t\ $ (true posterior).

Predicted clean image (Tweedie): $\ \hat x_0 = \frac{1}{\sqrt{\bar\alpha_t}}\big(x_t - \sqrt{1-\bar\alpha_t}\,\epsilon_\theta(x_t,t)\big)$.

Learned interpolated variance: $\ \sigma_t^2 = \exp\!\big(v\log\beta_t + (1-v)\log\tilde\beta_t\big),\ v=v_\theta(x_t,t)\in[0,1]$.

Algorithm 2 — Sampling. Draw $x_T\sim\mathcal N(0,I)$; for $t=T,\dots,1$ set $z\sim\mathcal N(0,I)$ if $t\gt 1$ else $z=0$, apply the boxed update; return $x_0$.

9 · DDIM & the score view (Lesson 9)

Non-Markovian forward family (the DDIM construction):

$$ q_\sigma(x_{t-1}\mid x_t,x_0)=\mathcal N\!\Big(\sqrt{\bar\alpha_{t-1}}\,x_0+\sqrt{1-\bar\alpha_{t-1}-\sigma_t^2}\;\frac{x_t-\sqrt{\bar\alpha_t}\,x_0}{\sqrt{1-\bar\alpha_t}},\;\sigma_t^2 I\Big). $$

DDIM reverse update:

$$ \boxed{\, x_{t-1} = \sqrt{\bar\alpha_{t-1}}\;\underbrace{\left(\frac{x_t - \sqrt{1-\bar\alpha_t}\,\epsilon_\theta(x_t,t)}{\sqrt{\bar\alpha_t}}\right)}_{\hat x_0 \text{ (predicted clean image)}} \;+\; \underbrace{\sqrt{1-\bar\alpha_{t-1}-\sigma_t^2}\;\epsilon_\theta(x_t,t)}_{\text{direction pointing to } x_t} \;+\; \sigma_t z \,} $$

Deterministic DDIM ($\sigma_t=0$): $\ x_{t-1}=\sqrt{\bar\alpha_{t-1}}\,\hat x_0+\sqrt{1-\bar\alpha_{t-1}}\;\epsilon_\theta(x_t,t)$. Interpolate with $\ \sigma_t=\eta\,\sqrt{\tilde\beta_t}$.

Score ↔ noise identity — the bridge to score-based models:

$$ \boxed{\ \nabla_{x_t}\log q_t(x_t) \approx -\frac{\epsilon_\theta(x_t,t)}{\sqrt{1-\bar\alpha_t}}\ } $$

Tweedie’s formula: $\ \mathbb E[\sqrt{\bar\alpha_t}\,x_0\mid x_t]=x_t+(1-\bar\alpha_t)\,\nabla_{x_t}\log q_t(x_t)$.

Annealed Langevin dynamics: $\ x \leftarrow x + \tfrac{\delta}{2}\,\nabla_x\log p(x) + \sqrt{\delta}\,z$.

Anderson reverse-time SDE: $\ dx=\big[f(x,t)-g(t)^2\,\nabla_x\log q_t(x)\big]dt+g(t)\,d\bar w$.

Probability-flow ODE (deterministic, shares the SDE’s marginals):

$$ \frac{dx}{dt} = f(x,t) - \tfrac{1}{2}g(t)^2\,\nabla_x\log q_t(x). $$

10 · Conditioning & guidance (Lesson 10)

Conditional training loss: $\ L = \mathbb{E}_{t,x_0,\epsilon,c}\big[\|\epsilon - \epsilon_\theta(x_t, t, c)\|^2\big]$.

Bayes decomposition of the conditional score:

$$ \nabla_{x_t}\log q_t(x_t\mid c) = \nabla_{x_t}\log q_t(x_t) + \nabla_{x_t}\log p(c\mid x_t). $$

Classifier guidance:

$$ \boxed{\ \hat\epsilon = \epsilon_\theta(x_t,t) - w\,\sqrt{1-\bar\alpha_t}\;\nabla_{x_t}\log p_\phi(c\mid x_t)\ } $$

Classifier-free guidance — extrapolate between conditional and unconditional predictions:

$$ \boxed{\ \hat\epsilon(x_t,t,c) = \epsilon_\theta(x_t,t,\varnothing) + w\,\big(\epsilon_\theta(x_t,t,c) - \epsilon_\theta(x_t,t,\varnothing)\big)\ } $$

Cross-attention conditioning:

$$ Q=\phi(x_t)W_Q\;(N\times d_k),\quad K=\tau(c)W_K\;(M\times d_k),\quad V=\tau(c)W_V\;(M\times d_v), $$

$$ \text{Attention}=\mathrm{softmax}\!\Big(\tfrac{QK^\top}{\sqrt{d_k}}\Big)V\;\in\mathbb R^{N\times d_v}. $$

Symbol glossary

Symbol	Meaning
$x_0$	clean data sample
$x_t$	noisy latent at step $t$; $x_T\approx\mathcal N(0,I)$
$T$	number of diffusion steps
$\beta_t$	forward variance schedule, $\beta_t\in(0,1)$
$\alpha_t$	$1-\beta_t$
$\bar\alpha_t$	$\prod_{s\le t}\alpha_s$ (cumulative signal retention)
$a_t,\,b_t$	shorthand $\sqrt{\bar\alpha_t}$ and $\sqrt{1-\bar\alpha_t}$
$q(\cdot)$	fixed forward / noising law
$p_\theta(\cdot)$	learned reverse / denoising law
$\epsilon$	standard Gaussian noise $\sim\mathcal N(0,I)$
$\epsilon_\theta$	network’s predicted noise
$\mu_\theta,\Sigma_\theta$	learned reverse mean and covariance
$\tilde\mu_t,\tilde\beta_t$	true posterior mean and variance
$\sigma_t$	sampling variance at step $t$
$\hat x_0$	network’s predicted clean image
$v$	the v-prediction target
$w$	guidance / loss-weight scale
$c$	conditioning signal (class, text)
$\varnothing$	the null / unconditional token
$\tau$	time (or condition) embedding vector
$D_{\mathrm{KL}}$	Kullback–Leibler divergence
$\nabla_{x}\log q(x)$	the score function
$f,\,g$	SDE drift and diffusion coefficients

---

← Appendix A: Acronyms · Course Index