Appendix C — Acronyms & Symbols
Technical course on encoder–decoder transformers — reference appendix.
A lookup table for every acronym and symbol used in the course, expanded in plain language. Keep it open alongside the lessons.
Acronyms
| Term | Expansion / meaning |
|---|---|
| seq2seq | Sequence-to-sequence: map an input sequence $x$ to an output sequence $y$ (translation, summarization). |
| enc–dec | Encoder–decoder: the two-stack architecture of this course (T5, BART). |
| MLM | Masked Language Modeling: BERT’s objective — mask ~15% of tokens, predict them in place (bidirectional, no generation). |
| CE | Cross-Entropy: the training loss, $-\sum_t\log\hat p_{t,y_t}$. |
| KV cache | Key/Value cache: stored keys/values to avoid recomputation at inference. Enc–dec has two — a growing self-attention cache and a fixed cross-attention cache. |
| FFN / MLP | Feed-Forward Network: the position-wise two-layer sublayer, $W_2\phi(W_1x+b_1)+b_2$, $d_{ff}\approx4d$. |
| MHA | Multi-Head Attention: $h$ parallel attention heads, $d_k=d/h$, recombined by $W_O$. |
| BPE / SentencePiece | Subword tokenizers; SentencePiece is T5’s. Reduce out-of-vocabulary pressure. |
| prefix-LM | Prefix Language Model: single stack with a bidirectional prefix + causal suffix mask (UniLM, GLM); the Lesson 6 middle ground. |
| MAP | Maximum A Posteriori: the highest-probability output sequence; beam search approximates it. |
| NMT | Neural Machine Translation. |
| BLEU | Bilingual Evaluation Understudy: precision-oriented n-gram MT metric with a brevity penalty. |
| ROUGE | Recall-Oriented Understudy for Gisting Evaluation: recall-oriented n-gram/LCS summarization metric. |
| LCS | Longest Common Subsequence (used by ROUGE-L). |
| KD | Knowledge Distillation: train a small student to imitate a large teacher. |
| OOV | Out-Of-Vocabulary: a token absent from the model’s vocabulary; motivated copy mechanisms. |
| LoRA | Low-Rank Adaptation: a parameter-efficient fine-tuning method. |
| RMSNorm | Root-Mean-Square Normalization: T5’s normalization (no mean subtraction). |
| GELU / ReLU / GeGLU | Activation functions used in the FFN (BART uses GELU; T5 ReLU, T5 v1.1 GeGLU). |
<bos> / <eos> / <pad> | Begin-of-sequence / end-of-sequence / padding special tokens. |
<X>,<Y>,<Z> | T5 sentinel tokens marking corrupted spans (Lesson 7). |
Symbols
| Symbol | Meaning |
|---|---|
| $x=(x_1,\dots,x_n)$ | Input (source) token sequence, length $n$. |
| $y=(y_1,\dots,y_m)$ | Output (target) token sequence, length $m$. |
| $\tilde x$ | Corrupted input used in denoising pretraining (Lesson 7). |
| $\theta\in\mathbb R^P$ | All learned parameters; $P$ the parameter count. |
| $d$ | Model (hidden) width. |
| $d_k,d_v$ | Key/query and value dimensions per head; usually $d_k=d_v=d/h$. |
| $h$ | Number of attention heads. |
| $N_{\text{enc}},N_{\text{dec}}$ | Number of encoder / decoder blocks (depths). |
| $d_{ff}$ | FFN inner width, $\approx 4d$. |
| $V,V_x,V_y$ | Vocabulary size (shared / source / target). |
| $E,W_U$ | Embedding matrix; unembedding (often $W_U=E^\top$). |
| $H_{\text{enc}}\in\mathbb R^{n\times d}$ | Encoder output memory — one contextual vector per input token. [STATE]. |
| $H_{\text{dec}}\in\mathbb R^{m\times d}$ | Decoder hidden states. |
| $W_Q,W_K,W_V,W_O$ | Attention projection matrices. Superscript $s$ = decoder self-attention; $c$ = cross-attention. |
| $Q,K,V$ | Query / key / value matrices (queries from decoder, keys/values from encoder, in cross-attention). |
| $M,M^{\text{causal}},M^{\text{pad}},M^{\text{prefix}}$ | Additive attention masks (0 or $-\infty$): causal, padding, prefix-LM. |
| $A$ | Attention matrix; square ($n\times n$ or $m\times m$) for self-attention, rectangular ($m\times n$) for cross-attention. |
| $s_{ij}$ | Attention score (scaled dot product) between query $i$ and key $j$. |
| $\mathbf z_t,\hat{\mathbf p}_t$ | Logits and predicted probability distribution at output position $t$. |
| $\mathbf y_t$ | One-hot target at position $t$. |
| $\Delta^{V-1}$ | Probability simplex over $V$ tokens (valid categorical distributions). |
| $\mathcal L$ | Loss (cross-entropy unless noted). |
| $T$ | Softmax temperature (inference, and distillation softening). |
| $k,\alpha$ | Beam width; length-penalty exponent (Lesson 9). |
| $\mathbf c^t$ | Coverage vector — cumulative attention to each source position (Lesson 9). |
| $p_{\text{gen}}$ | Pointer-generator gate: probability of generating vs. copying (Lesson 9). |
| $\lambda$ | Distillation mixing weight (Lesson 10). |
| $\sigma,\phi$ | Sigmoid; FFN activation. |
← Course index · Appendix A — Formulas · Appendix B — Architecture Comparison