Appendix B — Cheat Sheets: How the Architectures Compare

This appendix is a reference. Where the chapters told the story of each architecture, here we lay two of the most important contrasts side by side, in plain comparisons you can return to when you need to decide which tool fits a job. Two comparisons matter most: the reader against the writer, and the dense model against the mixture-of-experts.

Reader versus writer

The one idea everything flows from

Every difference between a reader (an encoder, Chapter 5 ) and a writer (a decoder, Chapter 4 ) traces back to a single design choice: what is the model allowed to look at? A writer may only look backward, at what it has already produced or been given, never at what comes next. A reader may look in both directions at once, at the whole passage. That one rule cascades into everything else. Because the writer can only see backward, it can be trained to guess the next word and then actually produce text. Because the reader can see everything, it can build deep understanding — but it loses the ability to guess the next word, since the answer is sitting right there in front of it.

The differences at a glance

A few of these deserve a word. The reader does its work in a single pass, reading everything in parallel, which is why it is cheap and fast for understanding tasks. The writer must loop, producing text one word at a time, each new word requiring another pass — inherently slower. And the two are adapted in fundamentally different ways: a reader is steered by supervision (attach a head, train on labeled examples), while a writer is steered by behavior (describe the task, show a couple of examples, or shape it as in Chapter 10 ), often with no task-specific data at all.

Which should I reach for?

Walk through three questions.

First: do you need the system to produce new text? If no — you want to classify, tag, extract, search, or match — reach for a reader. Sentiment analysis, spam detection, search relevance, document tagging all live here. If yes, keep going.

Second: are you transforming one whole given input into an output — translating it, summarizing it, rewriting it? If yes, reach for a reader-plus-writer combination that reads the input deeply and then writes the output. If it is free-form instead — open chat, completions, answering questions — reach for a writer on its own.

Third: are you building search or a retrieval-augmented system? Then you will often want both: a reader to understand and match documents to a query, and a writer to compose the final answer from what was retrieved. The reader finds and understands; the writer speaks. They are complements, not competitors.

Figure 1. One rule, and everything that cascades from it.

Dense versus mixture-of-experts

The one idea everything flows from

A dense model uses all of itself to process every word. A mixture-of-experts model (Chapter 7 ) keeps a large bank of expert sub-modules but, for each word, switches on only a small handful, chosen by a little router. That single change — “use everything every time” versus “use a chosen few each time” — drives every other difference.

The differences at a glance

The headline is the two size numbers. A dense model has one: how much it knows and how much work it does per word are the same figure, welded together. A mixture-of-experts model splits them — a large total size that sets how much it can know and how much memory it occupies, and a small active size that sets how much work each word costs. This is the whole trick: it decouples “how much it knows” from “how much it costs to run per word,” so you can grow the total enormously while keeping the per-word cost modest. The catch is memory: all the experts must stay loaded even though each word uses only a few, so a mixture-of-experts model has a small model’s compute but a large model’s memory.

The router is the new piece, and it brings problems a dense model never faces: the lazy-favoritism collapse, the consequent need for active load balancing, more knobs to set, and extra fragility that demands careful handling. It is also worth stressing what stays identical: the attention mechanism, the caching used during generation, and all the dials that control output. The entire mixture-of-experts trick lives inside one sub-layer; from the outside, the model behaves like any other.

When should I reach for a mixture-of-experts model?

Start with memory, because it is the deciding factor. Is memory tight — do you need to fit on just a few devices? If memory is your binding constraint, stay dense: a mixture-of-experts model would force you to hold all the experts in memory anyway, so its main advantage (cheap per-word compute) does not help you when memory is exactly what you have run out of. With a dense model, your capacity is simply whatever you can hold.

If you have memory to spare, ask: do you want more quality without paying more compute per word? If yes, this is the sweet spot for a mixture-of-experts model — spend memory to buy extra capacity and quality while keeping the per-word running cost flat. If instead simplicity matters more to you, stay dense and avoid the router, the balancing machinery, and the cross-machine shuffling entirely.

The compressed rule: reach for a mixture-of-experts model when you have memory to spend and want higher quality at low per-word cost; stay dense when memory is the binding constraint, or when operational simplicity is worth more than the compute savings.

Figure 2. Use everything every time, or use a chosen few each time.

Figure 3. Start with memory, because it is the deciding factor.

← Appendix A · Field Guide to Real Models · Contents · Appendix C · Attention Variants →

The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.