How Encoder–Decoder LLMs Work

A self-contained, technically rigorous course on the mechanics of encoder–decoder (sequence-to-sequence) transformers — the T5 / BART family — told as the explicit bridge between a decoder-only LLM course and the deeper architecture literature. It assumes you already know what a transformer and self-attention are, and spends its time on the nuances the encoder-only and decoder-only stories leave out.

The course tells one story. A sequence-to-sequence model is a conditional function $f(y_{\lt t};x,\theta)$ that generates an output sequence $y$ conditioned on an input sequence $x$. Lesson 1 frames why that conditioning wants two stacks instead of one. Lessons 2–5 build the two stacks bottom-up — the bidirectional encoder, the causal decoder, the cross-attention that bonds them, and the full encoder–decoder block. Lesson 6 shows the prefix-LM is the same object with the seam removed. Lessons 7–8 cover training; Lesson 9 covers inference; Lesson 10 fine-tunes and distills the result.

The encoder compresses context; the decoder generates conditioned on it; cross-attention is the only new mechanism.

Audience: comfortable with decoder-only LLMs at a high level; new to the encoder–decoder specifics. Prerequisites: linear algebra, multivariate calculus, basic probability, and a first pass through decoder-only transformer mechanics. The LLM course is a natural companion.

Appendices