Appendix A — Formula Sheet
Technical course on encoder–decoder transformers — reference appendix.
A lookup table of every key equation, collected by lesson, with a one-line reminder of what each says. The cross-attention shapes (Lesson 4) are the ones worth memorizing. Keep this open alongside the lessons.
Lesson 1 — The seq2seq problem
Conditional factorization — generate the output autoregressively, conditioned on the whole input:
$$ P_\theta(y\mid x)=\prod_{t=1}^{m} P_\theta\big(y_t\mid y_{\lt t},\,x\big). $$The two maps — encoder runs once, decoder runs per output token:
$$ \mathrm{Enc}: x\mapsto H_{\text{enc}}\in\mathbb R^{n\times d},\qquad \mathrm{Dec}: (y_{\lt t},H_{\text{enc}})\mapsto \hat{\mathbf p}_t\in\Delta^{V_y-1}. $$Lesson 2 — Encoder (bidirectional self-attention)
No causal mask (padding only):
$$ \mathrm{SelfAttn}_{\text{bidir}}(H)=\mathrm{softmax}\!\Big(\tfrac{QK^\top}{\sqrt{d_k}}+M^{\text{pad}}\Big)V, \quad Q,K,V=HW_Q,HW_K,HW_V,\quad M^{\text{enc}}_{ij}=0. $$Encoder block (two pre-norm residual sublayers):
$$ H'=H+\mathrm{MHA}_{\text{bidir}}(\mathrm{Norm}(H)),\qquad H^{\text{out}}=H'+\mathrm{FFN}(\mathrm{Norm}(H')). $$Lesson 3 — Decoder (causal self-attention)
Causal mask:
$$ M^{\text{causal}}_{ij}=\begin{cases}0,& j\le i\\ -\infty,& j\gt i\end{cases} $$Decoder block (three pre-norm residual sublayers):
$$ \begin{aligned} H_1 &= H + \mathrm{SelfAttn}_{\text{causal}}(\mathrm{Norm}(H))\\ H_2 &= H_1 + \mathrm{CrossAttn}(\mathrm{Norm}(H_1),H_{\text{enc}})\\ H_3 &= H_2 + \mathrm{FFN}(\mathrm{Norm}(H_2)) \end{aligned} $$Output head: $\ \mathbf z_t=\mathbf h_t^\top W_U,\quad \hat{\mathbf p}_t=\mathrm{softmax}(\mathbf z_t),\quad W_U=E^\top$ (tied).
Lesson 4 — Cross-attention (the only new mechanism)
Two sources — queries from decoder, keys/values from encoder:
$$ Q=H_{\text{dec}}W_Q^c\in\mathbb R^{m\times d_k},\quad K=H_{\text{enc}}W_K^c\in\mathbb R^{n\times d_k},\quad V=H_{\text{enc}}W_V^c\in\mathbb R^{n\times d_v}. $$Rectangular attention ($m$ output rows, $n$ input columns), source padding mask, no causal mask:
$$ A=\mathrm{softmax}\!\Big(\tfrac{QK^\top}{\sqrt{d_k}}+M^{\text{pad}}\Big)\in\mathbb R^{m\times n},\qquad \mathrm{CrossAttn}=AV\in\mathbb R^{m\times d_v}. $$$$ \mathbf o_i=\sum_{j=1}^{n}A_{ij}\mathbf v_j,\qquad \textstyle\sum_{j=1}^{n}A_{ij}=1\ \ \forall i. $$Gradient into the encoder (the learning bridge):
$$ \frac{\partial\mathcal L}{\partial H_{\text{enc}}}=\frac{\partial\mathcal L}{\partial K}\frac{\partial K}{\partial H_{\text{enc}}}+\frac{\partial\mathcal L}{\partial V}\frac{\partial V}{\partial H_{\text{enc}}}. $$Lesson 5 — Full architecture & parameter budget
$$ P\approx 12\,N_{\text{enc}}\,d^2 + 16\,N_{\text{dec}}\,d^2 + Vd, \quad\text{(enc block }12d^2,\ \text{dec block }16d^2,\ \text{cross-attn }+4d^2). $$T5 relative-position bias (added to logits, no input position embedding):
$$ s_{ij}=\frac{\mathbf q_i^\top\mathbf k_j}{\sqrt{d_k}}+b_{\,\text{bucket}(i-j)}. $$Lesson 6 — Prefix-LM mask
$$ M^{\text{prefix}}_{ij}=\begin{cases} 0,& j\le n\ (\text{bidirectional prefix})\\ 0,& n\lt j\le i\ (\text{causal suffix, past})\\ -\infty,& j\gt i,\ j\gt n\ (\text{future suffix}) \end{cases} $$Lesson 7 — Denoising objectives
Span corruption (T5): spans → sentinels in $\tilde x$; decoder emits the missing spans. BART: mask / delete / infill / permute sentences / rotate document; decoder rebuilds the whole document.
Loss (same for both, and for fine-tuning):
$$ \mathcal L(\theta)=-\frac{1}{|y|}\sum_{t}\log P_\theta(y_t\mid y_{\lt t},\tilde x). $$Lesson 8 — Teacher forcing
Shifted decoder input / targets:
$$ \text{input}=(\langle\text{bos}\rangle,y_1,\dots,y_{m-1}),\qquad \text{targets}=(y_1,\dots,y_m). $$Logit gradient (predicted − target):
$$ \frac{\partial\mathcal L}{\partial\mathbf z_t}=\hat{\mathbf p}_t-\mathbf y_t. $$Lesson 9 — Inference
Cross-attention KV cache — computed once from $H_{\text{enc}}$, fixed for all steps; self-attention cache grows by one per step.
Length-normalized beam score:
$$ \text{score}(y)=\frac{1}{\mathrm{lp}(m)}\sum_{t=1}^{m}\log\hat p_{y_t},\qquad \mathrm{lp}(m)=\frac{(5+m)^\alpha}{6^\alpha}. $$Coverage vector / loss:
$$ \mathbf c^t=\sum_{t'\lt t}A_{t',:},\qquad \mathcal L_{\text{cov}}=\sum_t\sum_j \min(A_{tj},c^t_j). $$Pointer-generator copy:
$$ P(w)=p_{\text{gen}}P_{\text{vocab}}(w)+(1-p_{\text{gen}})\!\!\sum_{j:x_j=w}\!\!A_{tj},\qquad p_{\text{gen}}\in[0,1]. $$Lesson 10 — Evaluation & distillation
BLEU (precision + brevity penalty):
$$ \text{BLEU}=\min(1,e^{1-r/c})\cdot\exp\!\Big(\textstyle\sum_{n=1}^{4}w_n\log p_n\Big). $$Distillation loss (hard labels + softened teacher):
$$ \mathcal L_{\text{KD}}=(1-\lambda)\,\mathrm{CE}(y,\hat{\mathbf p}^{\,s})+\lambda T^2 D_{\mathrm{KL}}\!\big(\hat{\mathbf p}^{\,t}_T\|\hat{\mathbf p}^{\,s}_T\big). $$← Course index · Appendix B — Architecture Comparison · Appendix C — Acronyms