Appendix A — A Field Guide to Real Models

The main chapters described the kinds of model in the abstract. This appendix puts faces to the families: the real, named systems you will actually hear about, and the one specific thing each one contributed. The most useful way to read a family tree like this is not as a list of products but as a story of fixes — each model usually exists because someone looked at the previous one, named a single flaw, and corrected it. Read this way, the intimidating roll-call of names becomes a short, sensible history.

The reader family: BERT and its descendants

These are the “reading” models from Chapter 5 — they take in a whole passage at once, in both directions, and turn it into understanding rather than new text. The entire family is best understood as a series of repairs to one founding model.

BERT (2018) — the original reader. BERT was the breakthrough that taught a machine to read a sentence by looking at the whole thing at once, left and right together, instead of word by word in one direction. It learned by playing fill-in-the-blank: hide some words and make the model recover them from context. That single combination — read both ways, learn by guessing hidden words — became the foundation of nearly every understanding task. But BERT had real flaws, and its children would pick them off one at a time: it wasn’t trained hard or long enough; one of its two training games was useless; only about one word in seven was actually teaching it anything on each pass; it couldn’t handle long documents; and much of its bulk was wasted on bookkeeping.

RoBERTa — “BERT was just undercooked.” RoBERTa changed nothing about the design. It simply trained the same model properly: far more text, much longer, with bigger batches, and it threw out the useless second training game (a “do these two sentences follow each other?” task that taught almost nothing). It also reshuffled which words got hidden each time rather than fixing them once. The startling result — same model, dramatically better — proved the original had been badly undertrained. (This is the episode discussed in Chapter 5: sometimes the cheapest big win is simply more and longer.)

ALBERT — “the model is bloated; slim it down.” ALBERT noticed BERT carried enormous weight in its vocabulary lookup and in repeating similar machinery layer after layer. It compressed the vocabulary storage and reused the same internal machinery across all its layers instead of keeping a fresh copy at each one, yielding a far smaller model. It also swapped the useless sentence game for a smarter one — telling whether two sentences are in the right order or flipped — which genuinely teaches coherence. The tradeoff: lighter to store, but the heavy reuse can make it slower to learn.

DistilBERT — “make it small enough to actually ship.” This one is about deployment, not cleverness. It trained a small “student” to imitate a full-size BERT “teacher,” copying not just the right answers but the teacher’s whole way of leaning toward them. The payoff: about forty percent smaller and sixty percent faster while keeping roughly ninety-seven percent of the quality. It is the standard move whenever you need a big model’s smarts on a phone or a budget.

ELECTRA — “stop wasting most of every sentence.” ELECTRA’s insight was that BERT only learned from the handful of words it hid; the rest of the sentence was just scenery. ELECTRA changed the game: a small helper swaps some words for plausible fakes, and the main model must look at every single word and call it real or fake. Because every word is now a teaching moment rather than one in seven, ELECTRA learns far faster, reaching BERT’s quality at roughly a quarter of the cost.

DeBERTa — “the way it tracks position is clumsy.” Earlier models mashed together two different things about each word — what it is and where it sits — into one blended signal. DeBERTa kept them separate, letting the model reason about meaning and position independently, and reinjected precise position information right where it matters most. This cleaner handling of “what versus where” kept it atop the accuracy leaderboards for a long stretch, making it a go-to when you want maximum precision.

ModernBERT (2024) — “give the reader everything the chatbot era figured out.” While the reader family slept, the text-writing models invented a pile of engineering upgrades. ModernBERT is the catch-up: it imports the modern way of tracking position (so it can read documents of eight thousand words rather than choking at five hundred), faster and steadier internal plumbing, and training that includes computer code. The result is a drop-in replacement for BERT that is faster and handles far longer documents — while keeping the original read-both-ways, fill-in-the-blank soul.

The through-line is clean: every member fixes one named flaw of BERT — undertraining (RoBERTa), bloat (ALBERT), shipping size (DistilBERT), wasted words (ELECTRA), clumsy positions (DeBERTa), and short memory plus dated engineering (ModernBERT).

Figure 1. The reader family, read as a story of fixes.

The mixture-of-experts family

These are the “committee of specialists” models from Chapter 7 : rather than running every word through one enormous brain, they keep a large bank of experts and, for each word, wake only the few best suited to it. The recurring villain in this family’s history is the router — the traffic cop that decides which experts each word visits — which, left alone, gets lazy and funnels everything to a few favorites. Most of the story is about taming it.

Sparsely-gated mixture-of-experts (2017) — the origin. This is the paper that introduced the whole committee idea, with a router that adds a little randomness to spread the load and the first “fairness penalties” to stop it playing favorites. It proved you could train thousands of specialists and use only a few at a time, and it named the two problems every successor would wrestle with: lazy favoritism and the need for balancing.

GShard (2020) — “make it work across many machines.” A committee this big cannot fit on one chip, so GShard worked out how to spread the experts across many machines and shuttle each word to the right one efficiently. It set the standard plumbing — split the experts across devices, cap how many words any one expert handles, penalize imbalance — that every later model inherited.

Switch Transformer (2021) — “send each word to just one expert.” Everyone had assumed each word needed at least two specialists for safety. Switch boldly cut it to one — the simplest possible setup — and showed that with the right guardrails it works fine and runs much faster. It made the whole approach feel robust rather than fragile.

GLaM (2021) — “prove the savings at giant scale.” GLaM was a clean, very large demonstration that a model holding over a trillion parameters’ worth of knowledge could answer each word using only a tiny fraction of itself — delivering the cost-versus-quality bargain at a scale that turned heads.

ST-MoE (2022) — “stop it from blowing up or overfitting.” Big committees were notoriously unstable to train and hard to adapt afterward. ST-MoE added a guardrail to keep the router’s decisions from spiking out of control, plus a careful recipe for fine-tuning — fixing the two practical headaches that had made large sparse models painful to use.

Mixtral (2023) — “the open breakout.” Mixtral made the approach mainstream and freely available. With eight experts per layer and two used per word, it matched or beat much larger conventional models while costing a fraction to run, proving to the open community that the committee approach was not just a big-lab trick.

DeepSeek-MoE / V2 / V3 (2024) — “more, slimmer specialists, plus a few always-on ones.” DeepSeek refined the recipe two ways. Instead of a few wide experts, it used many slim ones, so each could specialize more sharply — plus a couple of “always-on” generalists every word visits for the common basics. And it dropped the fairness penalty (which slightly distorted the model’s real goal) in favor of a gentle nudge that keeps the router balanced without polluting the actual learning. Its largest version scales to hundreds of experts while running only a small slice per word.

These crystallized into two house styles: the Mixtral style (a few wide experts, kept fair with a penalty) is simple and dependable; the DeepSeek style (many slim experts plus always-on generalists, kept fair with a gentle nudge) squeezes out more specialization and efficiency at the cost of more moving parts. Later families such as Qwen’s mixture-of-experts models largely follow the slim-plus-generalist pattern, confirming it as a modern default.

Figure 2. The committee family, and the router it kept taming.

The sequence-to-sequence zoo, and how the three body plans map to famous names

Step all the way back and there are really just three body plans, all built from the same parts but wired differently. Readers (encoder-only) take in a whole input both directions and produce understanding; they do not write. Writers (decoder-only) read left to right and keep going, predicting the next word over and over; this is the design behind essentially every modern chatbot, and it handles translation-style jobs by simply pasting the input in front and continuing. Reader-writers (encoder–decoder) bolt the two together with the cross-attention bridge from Chapter 6 , and are the natural fit when input and output are clearly separate things of different lengths — translation, summarization, speech-to-text.

The rule of thumb: read both directions for understanding; read one direction and continue for generation; do both with a bridge for transduction — turning one whole thing into another.

A little history explains why one plan came to dominate. From roughly 2017 to 2020, reader-writers ruled translation and readers ruled understanding. But from 2020 on, the pure writer design took over the general-purpose world, because one uniform left-to-right stack is the simplest thing to scale and the easiest to steer with a prompt. Reader-writers survive where their built-in strengths still pay off.

Here are the famous faces, sorted by body plan:

• The original Transformer (2017) — the reader-writer prototype, built to translate, that started everything.
• BERT, RoBERTa — the readers: understanding and search, no writing.
• GPT-2, GPT-3, Llama, Mistral — the writers: the chatbot and general-purpose lineage everyone knows, doing translation-style tasks just by pasting input in front.
• T5 (and mT5, Flan-T5) — reader-writers that reframed everything as “text in, text out,” so one model could translate, summarize, and answer with the same setup.
• BART (and mBART) — reader-writers trained by deliberately corrupting text and learning to repair it; especially strong at summarization.
• Pegasus — a reader-writer purpose-built for summarization, trained by hiding whole key sentences and regenerating them.
• UniLM, GLM — the clever middle ground: a single stack that reads its input both directions but writes its output one direction, getting some reader-writer benefit without a separate second half.
• Whisper — a reader-writer for audio: one half listens, the other writes the transcript, with the bridge connecting ear to pen.
• Flamingo and many vision-language models — writers with a bridge to a vision system, letting a text model “look at” an image and describe it.

The closing point is the one worth keeping: that bridge between reader and writer is not merely a text trick. The moment you see it as “let the thing being written constantly consult the thing being understood,” it generalizes everywhere — text to text, speech to text, image to text. It is the universal bond whenever one stream of information must guide the creation of another.

Figure 3. Three body plans, and the famous names that fit each.

← Chapter 12 · The Whole Landscape · Contents · Appendix B · Comparison Cheat Sheets →

The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.