How Encoder–Decoder LLMs Work on Ship of Theseus

Why Two Stacks? The Sequence-to-Sequence Problem

Mon, 01 Jan 0001 00:00:00 +0000

Lesson 1 — Why Two Stacks? The Sequence-to-Sequence Problem A 10-lesson technical course on encoder–decoder (seq2seq) transformers. Audience: you already know the decoder-only transformer — self-attention, next-token prediction, softmax cross-entropy — and want the encoder–decoder nuances. Prerequisites: linear algebra, multivariate calculus, basic probability. We reuse standard notation and define what is new as we go. Course table of contents # Lesson Core technical content 1 Why Two Stacks? (this file) The conditional map $P_\theta(y\mid x)$; encoder-only vs. decoder-only vs. encoder–decoder; why seq2seq wants two stacks 2 The Encoder Stack Bidirectional self-attention (no mask); $H_{\text{enc}}\in\mathbb R^{n\times d}$ as fixed memory; padding masks; vs. BERT 3 The Decoder Stack Causal self-attention revisited; the three-sublayer block; what is added vs. a decoder-only model 4 Cross-Attention $Q=H_{\text{dec}}W_Q$, $K,V=H_{\text{enc}}W_{K},H_{\text{enc}}W_V$; the non-square $m\times n$ map; gradient into the encoder 5 The Encoder–Decoder Block Self-attn + cross-attn + FFN; full architecture; parameter budget; T5 vs. BART 6 Prefix-LM The prefix mask; folding two stacks into one; the encoder-only ↔ decoder-only spectrum 7 Pretraining Objectives T5 span corruption with sentinels; BART’s five noises; vs. MLM and next-token CE 8 Training: Teacher Forcing Shifted decoder inputs; CE over target positions; parallel training; gradient through cross-attention; exposure bias 9 Inference Encode-once/decode-many; the two KV caches; beam search; coverage and copy mechanisms 10 Fine-Tuning & Distillation Text-to-text framing; translation/summarization; BLEU/ROUGE; distilling T5-small A Appendix A — Formulas All key equations, collected by lesson B Appendix B — Architecture Comparison The three families along masking, conditioning, objective, use-case axes C Appendix C — Acronyms Every acronym and symbol, expanded One invariant runs through every lesson and is worth stating before anything else:

Read the full post →

The Encoder Stack: Bidirectional Context

Mon, 01 Jan 0001 00:00:00 +0000

Lesson 2 — The Encoder Stack: Bidirectional Context Technical course on encoder–decoder transformers — Lesson 2 of 10. Recap of Lesson 1 A seq2seq model is the conditional map $P_\theta(y\mid x)=\prod_t P_\theta(y_t\mid y_{\lt t},x)$: the input $x$ is fully available, the output $y$ is generated left to right. That asymmetry motivates two stacks — a bidirectional encoder that reads $x$ once into a memory $H_{\text{enc}}\in\mathbb R^{n\times d}$ (which is [STATE], recomputed per input), and a causal decoder that generates conditioned on it via cross-attention, the one genuinely new mechanism. We now build the first stack and make precise what dropping the causal mask buys.

Read the full post →

The Decoder Stack: Causal Self-Attention Revisited

Mon, 01 Jan 0001 00:00:00 +0000

Lesson 3 — The Decoder Stack: Causal Self-Attention Revisited Technical course on encoder–decoder transformers — Lesson 3 of 10. Recap of Lesson 2 The encoder reads $x$ with bidirectional self-attention ($M=0$) and emits the memory $H_{\text{enc}}\in\mathbb R^{n\times d}$ — one contextual vector per input token, computed once, frozen for generation, and tagged [STATE]. It keeps a padding mask so pad positions are never attended to, and passes that mask downstream. Its only deviation from a decoder-only block was a deleted causal mask. We now build the writer, whose self-attention keeps the causal mask but which grows one extra sublayer to consume the memory.

Read the full post →

Cross-Attention: The Only New Mechanism

Mon, 01 Jan 0001 00:00:00 +0000

Lesson 4 — Cross-Attention: The Only New Mechanism Technical course on encoder–decoder transformers — Lesson 4 of 10. Recap of Lesson 3 The decoder block has three pre-norm residual sublayers: causal self-attention over the output so far, then cross-attention to the encoder memory, then an FFN. The first and last are inherited unchanged from a decoder-only model; the middle one is the single new operation in the whole architecture, and the decoder carries a second full set of attention projections to run it. We deferred its derivation twice. This lesson pays the debt: cross-attention is scaled dot-product attention with one change — its queries come from the decoder, its keys and values from the encoder.

Read the full post →

The Encoder–Decoder Block & Full Architecture

Mon, 01 Jan 0001 00:00:00 +0000

Lesson 5 — The Encoder–Decoder Block & Full Architecture Technical course on encoder–decoder transformers — Lesson 5 of 10. Recap of Lesson 4 Cross-attention is scaled dot-product attention with queries from the decoder and keys/values from the encoder memory: $Q=H_{\text{dec}}W_Q^c$, $K,V=H_{\text{enc}}W_K^c,H_{\text{enc}}W_V^c$, giving a rectangular alignment $A\in\mathbb R^{m\times n}$ and output $AV\in\mathbb R^{m\times d_v}$. It uses the source padding mask but no causal mask, and it is the sole bridge for the training gradient into the encoder. With all three decoder sublayers derived, we now assemble the full model, count its parameters, and locate where T5 and BART deviate from the textbook Transformer.

Read the full post →

Prefix-LM: Encoder–Decoder as a Special Case

Mon, 01 Jan 0001 00:00:00 +0000

Lesson 6 — Prefix-LM: Encoder–Decoder as a Special Case Technical course on encoder–decoder transformers — Lesson 6 of 10. Recap of Lesson 5 The full model is a stack of two-sublayer encoder blocks and three-sublayer decoder blocks; the encoder runs once into $H_{\text{enc}}$, broadcast to every decoder block’s cross-attention. Parameters scale as $12N_{\text{enc}}d^2+16N_{\text{dec}}d^2+Vd$. T5 and BART differ from the textbook only in [CONFIG] dressings (positions, norm, activation, objective), never in the cross-attention core. We now zoom out: the two-stack design is one point on a spectrum, and a single mask connects it to the decoder-only model.

Read the full post →

Pretraining Objectives: Span Corruption vs. Denoising

Mon, 01 Jan 0001 00:00:00 +0000

Lesson 7 — Pretraining Objectives: Span Corruption vs. Denoising Technical course on encoder–decoder transformers — Lesson 7 of 10. Recap of Lesson 6 The encoder–decoder is one point on a masking spectrum: a prefix-LM folds the two stacks into one with a block mask (bidirectional prefix, causal suffix), the decoder-only sits at the fully-causal end, and encoder-only at the fully-bidirectional end. All differ only in mask and in whether $K,V$ come from a second sequence. T5’s ablation found the two-stack design strongest for seq2seq transfer. We now enter the training phase and ask: on what objective is a fresh encoder–decoder pretrained? The answer — denoising — is where T5 and BART diverge most from both MLM and next-token prediction.

Read the full post →

Training: Teacher Forcing & the Loss

Mon, 01 Jan 0001 00:00:00 +0000

Lesson 8 — Training: Teacher Forcing & the Loss Technical course on encoder–decoder transformers — Lesson 8 of 10. Recap of Lesson 7 Encoder–decoders are pretrained by denoising: corrupt a document into $\tilde x$, encode it bidirectionally, and have the decoder autoregressively reconstruct the clean/missing content $y$. T5 uses span corruption (sentinels, short targets); BART mixes five noises and rebuilds the whole document. Either way the loss is cross-entropy over decoder targets with the gradient $\hat{\mathbf p}-\mathbf y$. We now make the training procedure precise — how the decoder is fed and scored — and draw the sharp line between how the model is trained and how it later generates.

Read the full post →

Inference: Decoding Against a Frozen Memory

Mon, 01 Jan 0001 00:00:00 +0000

Lesson 9 — Inference: Decoding Against a Frozen Memory Technical course on encoder–decoder transformers — Lesson 9 of 10. Recap of Lesson 8 Training uses teacher forcing: the decoder is fed the ground-truth output shifted right by one, so all $m$ positions are scored in parallel under a causal mask, with cross-attention reading the full encoder memory. The loss is averaged cross-entropy ($\hat{\mathbf p}_t-\mathbf y_t$ gradient), and that gradient is the encoder’s only learning signal, routed through cross-attention. The cost is exposure bias — the model trains on correct prefixes but must later condition on its own. We now run the frozen model: $\theta$ is fixed, no loss, no gradient, only forward passes — and every knob is [CONFIG], every buffer [STATE].

Read the full post →

Fine-Tuning, Task Framing & Distillation

Mon, 01 Jan 0001 00:00:00 +0000

Lesson 10 — Fine-Tuning, Task Framing & Distillation Technical course on encoder–decoder transformers — Lesson 10 of 10. Recap of Lesson 9 Inference encodes once, decodes many: the encoder builds $H_{\text{enc}}$ in one pass; the decoder generates autoregressively with two KV caches — a growing self-attention cache and a fixed cross-attention cache. Beam search with length normalization is the seq2seq default; the alignment matrix $A$ powers coverage (anti-repetition) and copy/pointer-generator mechanisms. $\theta$ is frozen, every knob [CONFIG], every buffer [STATE]. We now take the pretrained model and make it useful: task framing, fine-tuning, evaluation, and compression.

Read the full post →

Appendix A — Formula Sheet

Mon, 01 Jan 0001 00:00:00 +0000

Appendix A — Formula Sheet Technical course on encoder–decoder transformers — reference appendix. A lookup table of every key equation, collected by lesson, with a one-line reminder of what each says. The cross-attention shapes (Lesson 4) are the ones worth memorizing. Keep this open alongside the lessons. Lesson 1 — The seq2seq problem Conditional factorization — generate the output autoregressively, conditioned on the whole input: $$ P_\theta(y\mid x)=\prod_{t=1}^{m} P_\theta\big(y_t\mid y_{\lt t},\,x\big). $$The two maps — encoder runs once, decoder runs per output token:

Read the full post →

Appendix B — Architecture Comparison & Model Zoo

Mon, 01 Jan 0001 00:00:00 +0000

Appendix B — Architecture Comparison & Model Zoo Technical course on encoder–decoder transformers — reference appendix. Where this fits This appendix is the side-by-side that Lessons 1 and 6 build toward. The three transformer families are not three different inventions — they are the same machinery under different attention masks (and different sources for keys/values). This sheet lays them on one table along the axes that actually distinguish them, then names the real models so the abstractions have faces. Read it after Lesson 6, or keep it open from the start as a map.

Read the full post →

Appendix C — Acronyms & Symbols

Mon, 01 Jan 0001 00:00:00 +0000

Appendix C — Acronyms & Symbols Technical course on encoder–decoder transformers — reference appendix. A lookup table for every acronym and symbol used in the course, expanded in plain language. Keep it open alongside the lessons. Acronyms Term Expansion / meaning seq2seq Sequence-to-sequence: map an input sequence $x$ to an output sequence $y$ (translation, summarization). enc–dec Encoder–decoder: the two-stack architecture of this course (T5, BART). MLM Masked Language Modeling: BERT’s objective — mask ~15% of tokens, predict them in place (bidirectional, no generation). CE Cross-Entropy: the training loss, $-\sum_t\log\hat p_{t,y_t}$. KV cache Key/Value cache: stored keys/values to avoid recomputation at inference. Enc–dec has two — a growing self-attention cache and a fixed cross-attention cache. FFN / MLP Feed-Forward Network: the position-wise two-layer sublayer, $W_2\phi(W_1x+b_1)+b_2$, $d_{ff}\approx4d$. MHA Multi-Head Attention: $h$ parallel attention heads, $d_k=d/h$, recombined by $W_O$. BPE / SentencePiece Subword tokenizers; SentencePiece is T5’s. Reduce out-of-vocabulary pressure. prefix-LM Prefix Language Model: single stack with a bidirectional prefix + causal suffix mask (UniLM, GLM); the Lesson 6 middle ground. MAP Maximum A Posteriori: the highest-probability output sequence; beam search approximates it. NMT Neural Machine Translation. BLEU Bilingual Evaluation Understudy: precision-oriented n-gram MT metric with a brevity penalty. ROUGE Recall-Oriented Understudy for Gisting Evaluation: recall-oriented n-gram/LCS summarization metric. LCS Longest Common Subsequence (used by ROUGE-L). KD Knowledge Distillation: train a small student to imitate a large teacher. OOV Out-Of-Vocabulary: a token absent from the model’s vocabulary; motivated copy mechanisms. LoRA Low-Rank Adaptation: a parameter-efficient fine-tuning method. RMSNorm Root-Mean-Square Normalization: T5’s normalization (no mean subtraction). GELU / ReLU / GeGLU Activation functions used in the FFN (BART uses GELU; T5 ReLU, T5 v1.1 GeGLU). <bos> / <eos> / <pad> Begin-of-sequence / end-of-sequence / padding special tokens. <X>,<Y>,<Z> T5 sentinel tokens marking corrupted spans (Lesson 7). Symbols Symbol Meaning $x=(x_1,\dots,x_n)$ Input (source) token sequence, length $n$. $y=(y_1,\dots,y_m)$ Output (target) token sequence, length $m$. $\tilde x$ Corrupted input used in denoising pretraining (Lesson 7). $\theta\in\mathbb R^P$ All learned parameters; $P$ the parameter count. $d$ Model (hidden) width. $d_k,d_v$ Key/query and value dimensions per head; usually $d_k=d_v=d/h$. $h$ Number of attention heads. $N_{\text{enc}},N_{\text{dec}}$ Number of encoder / decoder blocks (depths). $d_{ff}$ FFN inner width, $\approx 4d$. $V,V_x,V_y$ Vocabulary size (shared / source / target). $E,W_U$ Embedding matrix; unembedding (often $W_U=E^\top$). $H_{\text{enc}}\in\mathbb R^{n\times d}$ Encoder output memory — one contextual vector per input token. [STATE]. $H_{\text{dec}}\in\mathbb R^{m\times d}$ Decoder hidden states. $W_Q,W_K,W_V,W_O$ Attention projection matrices. Superscript $s$ = decoder self-attention; $c$ = cross-attention. $Q,K,V$ Query / key / value matrices (queries from decoder, keys/values from encoder, in cross-attention). $M,M^{\text{causal}},M^{\text{pad}},M^{\text{prefix}}$ Additive attention masks (0 or $-\infty$): causal, padding, prefix-LM. $A$ Attention matrix; square ($n\times n$ or $m\times m$) for self-attention, rectangular ($m\times n$) for cross-attention. $s_{ij}$ Attention score (scaled dot product) between query $i$ and key $j$. $\mathbf z_t,\hat{\mathbf p}_t$ Logits and predicted probability distribution at output position $t$. $\mathbf y_t$ One-hot target at position $t$. $\Delta^{V-1}$ Probability simplex over $V$ tokens (valid categorical distributions). $\mathcal L$ Loss (cross-entropy unless noted). $T$ Softmax temperature (inference, and distillation softening). $k,\alpha$ Beam width; length-penalty exponent (Lesson 9). $\mathbf c^t$ Coverage vector — cumulative attention to each source position (Lesson 9). $p_{\text{gen}}$ Pointer-generator gate: probability of generating vs. copying (Lesson 9). $\lambda$ Distillation mixing weight (Lesson 10). $\sigma,\phi$ Sigmoid; FFN activation. ← Course index · Appendix A — Formulas · Appendix B — Architecture Comparison

Read the full post →