Chapter 7 — The Committee: Mixture-of-Experts

Every model we have met so far shares a hidden assumption: when it processes a word, it uses all of itself to do it. Every one of its billions of knobs participates in every word. This is so natural that it is easy to miss as an assumption at all — but it has an expensive consequence, and the family of models in this chapter exists to escape it. These are mixture-of-experts models, the family that includes Switch, Mixtral, and DeepSeek, and their central idea is to build a model that is vast in what it knows while only switching on a small part of itself for any given word.

The weld between size and cost

Start with the problem. In an ordinary model — called a dense model, because all of it is always engaged — capacity and cost are welded together. The only way to make such a model know more is to make it bigger, and making it bigger makes it proportionally slower and more expensive, because every newly added part now has to fire for every single word. Picture a vast factory where, to process one item, you must switch on every machine on the floor, including the ones with nothing to do with that item. Want more capability? Add more machines — but now they all run, all the time, for everything. There is no way to grow the factory’s abilities without growing its electricity bill in lockstep.

A mixture-of-experts model breaks that weld. Instead of one monolithic machine that handles everything, it keeps a whole bank of smaller specialists — called experts — and adds a small decision-maker — the router — that looks at each incoming word and sends it to just a couple of the most relevant experts. The other experts stay switched off for that word. The model can therefore hold an enormous amount of knowledge across all its experts while only doing a small amount of work on any given word.

The hospital

One analogy carries this entire chapter, so it is worth setting up carefully. Picture a large teaching hospital with hundreds of specialist doctors on staff. A patient — think of each word as a patient — arrives and is met by a triage nurse, who takes one look and sends them to the two specialists best suited to their case. The hospital’s total expertise is enormous, because every specialty is represented somewhere on staff. But no single patient consumes the whole hospital. They see two doctors and go home. The triage nurse is the router; the doctors are the experts; the patient is a word passing through. That is the whole idea. Everything else in this chapter is the practical machinery a hospital like that needs in order to actually function.

Two different sizes that everyone confuses

Once a model only uses part of itself, the single notion of “model size” splits into two numbers, and keeping them apart is essential. The first is the total number of parameters — everything the model stores, every expert whether or not it is used on a given word. This is the model’s capacity, the full breadth of what it can know. It is the size of the hospital’s entire staff. The second is the number of activated parameters — the much smaller slice actually put to work on any one word, namely the router plus the two-or-so experts it picked. This is what determines speed and running cost. It is the handful of doctors a single patient actually sees.

Concrete figures make the gap vivid. Mixtral stores around 47 billion parameters but only puts about 13 billion of them to work on each word. DeepSeek’s large model stores 671 billion but works with only about 37 billion at a time. The whole art of designing one of these models is buying a large total — more knowledge, better answers — while keeping the activated amount, and therefore the per-word cost, small. You get the staff of a giant hospital but pay, per patient, for a small clinic. A useful slogan to carry forward: cheap to compute, expensive to store. That asymmetry will reappear at every stage.

Figure 1. Two sizes: what a committee stores, and what one word uses.

What an expert is, and what the router does

There is nothing exotic inside an expert. Recall from Chapter 4 that each block of a model has a stage where every word is processed on its own, after attention has gathered its context — and that this stage is where most of the model’s stored knowledge lives. A mixture-of-experts model simply makes many copies of that stage — eight of them, or sixty-four, or more — each with its own independently learned settings, and calls each copy an expert. They all have the same shape but different “training,” like a roomful of doctors who attended the same medical school and then each specialized differently. The attention machinery is left completely untouched. The whole mixture-of-experts story lives in this one swapped-out stage.

The router is astonishingly small — a tiny sliver of the model — yet it controls everything. For each word, it produces a score for every expert, essentially answering “how much does this word want this expert?”, then picks the few highest-scoring ones and assigns each a weight reflecting its confidence. Only those chosen experts run; their outputs are blended together according to the router’s weights; the rest of the experts contribute nothing because they were never consulted. The triage nurse glances at the patient, picks the two best specialists, and sends everyone else back to their offices.

Some newer designs add refinements. Shared experts are a few that every word passes through no matter what, handling the common everyday processing all words need — the general intake exam everyone gets before being routed to specialists — so the chosen specialists don’t each have to relearn the basics. Fine-grained experts slice each expert thinner and use many more of them, giving the router far more combinations to choose from and room to specialize more precisely.

Figure 2. The triage nurse, and the two doctors the patient sees.

The deep difficulty: learning from a choice, and the lazy router

Here lies the central headache of the whole approach, and it is worth understanding because it explains most of what makes these models tricky. Recall from Chapter 1 that a model learns by nudging its knobs in whatever direction reduces its mistakes. But the router’s action is a choice — send this word to expert three, not expert seven — and a choice is all-or-nothing. You cannot send a word slightly to expert three. So the usual smooth learning signal has nothing to grab onto at the moment of selection. The router can be told “the experts you picked turned out well, lean into them” or “they turned out badly, trust them less,” but it never gets a clean signal saying “you should have picked a completely different expert,” because the expert it didn’t pick produced no result to compare against. The router learns to weight its winners well; it gets no clean nudge to change its winners.

This blind spot breeds a nasty failure called expert collapse. Suppose, just by luck at the start, a few experts are slightly better than the rest. The router sends them more words. More words means more practice, so they improve further. So the router sends them yet more. Meanwhile the neglected experts get almost no words, never improve, and fall further out of favor. The end state is a handful of experts doing nearly all the work while the rest sit idle as dead weight. This is disastrous, because capacity was the entire point: if you built a sixty-four-expert model but the router only ever uses four of them, you paid the storage bill for sixty-four and got the brains of four. In the hospital, the triage nurse keeps funneling every patient to the same two overworked doctors while the rest read magazines — you have staffed a huge hospital and run it like a tiny one, and the two favorites become an exhausted bottleneck.

Figure 3. The spiral that leaves most of the hospital reading magazines.

Load balancing: gently insisting the work be spread

Crucially, the model’s ordinary “get the answer right” goal does nothing to fix this, because a collapsed model can still produce reasonable answers using its few good experts. Balance has to be encouraged separately and deliberately. The fix is to add a second, mild goal alongside “be correct”: use all the experts roughly evenly. In plain terms, the model is given a nagging penalty whenever the router is overloading some experts and starving others, and that penalty pushes the router to ease off its favorites and give the neglected experts a turn. Over many rounds of training, the workload levels out. This balancing pressure — and the dials controlling how hard it pushes — is the single biggest way mixture-of-experts training differs from ordinary training. A dense model has nothing to balance, because it makes no choices. The newest designs achieve the same end even more cleanly, by keeping a running tally of how busy each expert is and quietly nudging the router toward idle ones — like a charge nurse rebalancing assignments on the fly, without ever overriding the doctors’ actual judgment.

Two more practical headaches

Because real hardware likes fixed, predictable batch sizes, each expert is given a fixed number of slots per batch — its capacity. But routing is unpredictable, so in any given batch some experts get swamped and others sit half-empty. If an expert is assigned more words than it has slots, the surplus words are simply dropped — they skip the expert and pass through unchanged, relying on the rest of the network to carry them along. There is a dial for how much headroom to give: too little and you turn patients away on busy days, too much and chairs sit empty. A well-balanced router drops very few words, and the occasional drop is survivable, but heavy dropping is a warning sign that balancing has gone wrong.

The second headache is communication. Because a mixture-of-experts model stores so many experts, they don’t fit on one machine, so they are spread across many — each machine holding a few. But the words being processed are also spread across machines, and the expert a given word needs probably lives on a different machine than the word does. So before the experts can run, every machine has to ship its words out to wherever the right experts are, and afterward ship all the results back. Every machine talks to every other machine, twice, for every such layer. Imagine the hospital split across a dozen buildings, with patients constantly shuttled to whichever building has their specialist and then shuttled home — and the shuttling, not the doctoring, becoming the slow part. This heavy, unavoidable communication is the defining cost of training these models, and it is why balance matters twice over: a layer can only finish as fast as its busiest expert, so an imbalanced router leaves expensive machines idle, waiting at the shuttle stop for the slowest one.

A reality check on “experts,” and the bargain that makes it worth it

It is tempting to imagine a tidy “French expert” and a “poetry expert” and a “math expert.” Reality is mushier. Specialization is real but weak and low-level: experts tend to react to surface patterns — punctuation, numbers, word types — more than to clean human topics, and responsibility for any given subject is usually smeared across several experts. The gains come less from crisp specialists than from the sheer extra capacity and the router’s freedom to mix and match a different small combination of experts for every word.

And the gains are real, which is the whole justification for the complexity. At a fixed amount of computing work per word, adding more experts reliably makes the model better. In other words, mixture-of-experts lets you convert a storage budget into quality without spending more compute budget. A dense model has only one lever — spend more compute, get more of everything, and pay for all of it on every word. A mixture-of-experts model has two levers, and can buy capacity with comparatively cheap storage instead of expensive computation. The returns shrink as you pile on more experts, and the extra storage and shuttling are genuine costs, so there is a sweet spot — but that two-lever bargain is why the entire enterprise is worthwhile.

Using one: a small model’s speed, a large model’s footprint

When the model is finally answering users, everything about how it generates text is exactly as it would be for a dense model, because none of that touches the experts. The one decisive difference is the asymmetry we have tracked all along, now in its most practical form: a mixture-of-experts model runs at the speed of a small model but demands the memory of a large one. You cannot simply keep the popular experts loaded and discard the rest, because routing is decided fresh for every word, so over any real stretch of text every expert eventually gets called. You must keep the whole hospital staffed and on-site, even though each patient sees only two doctors. This flips the usual bottleneck: a dense model is limited by raw computation, while a mixture-of-experts model is limited by memory — how much you have and how fast you can move it. The tricks for serving these models therefore all target memory, and we return to them in Chapter 11 .

What to carry forward

A mixture-of-experts model breaks the weld between how much a model knows and how much work it does per word. It replaces the single processing stage with a bank of experts and a tiny router that sends each word to just a few of them — so total capacity can be enormous while the per-word cost stays small. The price of this cleverness is a cluster of new problems all rooted in the router’s all-or-nothing choices: the lazy-router collapse that demands active load balancing, dropped words when experts overflow, and heavy machine-to-machine shuttling during training. And the defining trade in the end is memory for quality: cheap to compute, expensive to store.

That completes the family of language models — four shapes built from one toolkit. But not everything worth generating is language. In Part III we turn to a completely different way of creating, one that builds images not word by word but all at once, out of pure noise. Turn to Chapter 8 .

← Chapter 6 · The Translators · Contents · Chapter 8 · Diffusion Models →

The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.