Appendix B — Architecture Comparison & Model Zoo
Technical course on encoder–decoder transformers — reference appendix.
Where this fits
This appendix is the side-by-side that Lessons 1 and 6 build toward. The three transformer families are not three different inventions — they are the same machinery under different attention masks (and different sources for keys/values). This sheet lays them on one table along the axes that actually distinguish them, then names the real models so the abstractions have faces. Read it after Lesson 6, or keep it open from the start as a map.
1. The three families on four axes
| Axis | Encoder-only | Decoder-only | Encoder–decoder |
|---|---|---|---|
| Attention mask | bidirectional ($M=0$) | causal ($M$ lower-tri) | bidirectional encoder + causal decoder |
| Key/value source | self | self | self (within each stack) + cross (decoder reads encoder) |
| New mechanism | — | — | cross-attention |
| Conditioning on input | n/a (no generation) | implicit: input in the causal prefix | explicit: separate stack + cross-attention every decoder layer |
| Input read | bidirectionally | causally (left-to-right) | bidirectionally (by the encoder) |
| Generates output? | no — produces representations | yes (continue the stream) | yes (decoder, conditioned) |
| Pretraining objective | MLM (mask & predict in place) | next-token CE | denoising / span corruption |
| Parameters per layer | $12d^2$ | $12d^2$ | $12d^2$ (enc) / $16d^2$ (dec) |
| Output lengths | = input length | one stream | input $n$, output $m$, decoupled |
| Natural tasks | classification, NER, retrieval, embeddings | open-ended generation, chat, few-shot | translation, summarization, structured transduction |
The progression left-to-right across “attention mask” is the whole story: encoder-only is all bidirectional, decoder-only is all causal, and the encoder–decoder mixes the two by giving each regime its own stack and bonding them with cross-attention. The prefix-LM (Lesson 6) is the fourth point — a single stack with a block mask that is bidirectional over the prefix and causal over the suffix, conditioning implicitly through shared weights instead of cross-attention.
Figure 1. The three families, laid on one table.
2. When to reach for each
Encoder-only when the task is understanding a fixed input and the output is a label, a span, or a vector — sentiment, classification, named-entity recognition, sentence embeddings, retrieval. No generation, so no decoder, so it is cheap and the bidirectional reading is pure upside.
Decoder-only when the task is open-ended generation, when you want one uniform stack that scales cleanly, in-context/few-shot learning, and a single model that does many things via prompting. This is the dominant general-purpose LLM design today. Seq2seq tasks are handled by concatenation (input → output), trading the encoder’s bidirectional reading for simplicity.
Encoder–decoder when the task is a genuine transduction with a clearly separate input and output — machine translation, summarization, grammatical error correction, data-to-text, speech-to-text — especially when reading the input well (bidirectionally) is decisive and the output length differs from the input. The compute-once memory and per-layer cross-attention conditioning are worth the extra stack here.
The field’s trajectory: 2017–2020 was encoder–decoder (MT) and encoder-only (BERT-era understanding); 2020 onward saw decoder-only scale to general-purpose dominance because one uniform causal stack is simpler to scale and prompt. Encoder–decoders persist where their inductive bias pays — translation and summarization systems, and many multimodal models whose decoder cross-attends to encoded image/audio features (the cross-attention of Lesson 4 reused across modalities).
Figure 2. When to reach for each.
3. Model zoo
| Model | Family | Notable specifics |
|---|---|---|
| Original Transformer (Vaswani 2017) | enc–dec | the prototype; sinusoidal positions, post-norm, trained on WMT translation |
| BERT / RoBERTa | encoder-only | MLM pretraining, [CLS] pooling, no generation |
| GPT-2 / GPT-3 / Llama / Mistral | decoder-only | causal, next-token CE, seq2seq by concatenation |
| T5 / mT5 / T5 v1.1 / Flan-T5 | enc–dec | text-to-text framing, span corruption, relative position bias, RMSNorm/pre-norm |
| BART / mBART | enc–dec | denoising autoencoder (5 noises), close to original Transformer; strong at summarization |
| Pegasus | enc–dec | gap-sentence pretraining, purpose-built for summarization |
| UniLM / GLM | prefix-LM | single stack, prefix mask — the Lesson 6 middle ground |
| Whisper | enc–dec | audio encoder + text decoder; cross-attention bridges modalities |
| Flamingo / many VLMs | dec + cross-attn | text decoder cross-attends to a frozen vision encoder’s features |
The last two rows make the broader point: once you see cross-attention as “queries from one sequence, keys/values from another,” the encoder–decoder pattern generalizes far beyond text-to-text — any time one modality or sequence must condition the generation of another, cross-attention is the bond.
Figure 3. The abstractions, with faces.
← Course index · Appendix A — Formulas · Appendix C — Acronyms
The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.
