Almost every language model shares a quiet assumption: to process a single word, it uses all of itself. Every one of its billions of knobs fires for every word, whether that word is a comma or a rare medical term. This feels so natural it barely registers as a choice — but it carries a heavy price. The only way to make such a model know more is to make it bigger, and a bigger model is proportionally slower and costlier, because all of it runs, all the time, for everything. Mixture of Experts — usually shortened to MoE — is the family of models that breaks this bargain. Its idea is to build something vast in what it knows while only switching on a small slice of itself for any given word. This piece is about how that trick works, the surprisingly awkward problems it creates, and the tidy division of labour it does not deliver.
The weld between size and cost
Start with the problem it solves. An ordinary model is called dense, because all of it is always engaged. In a dense model, capacity and cost are welded together: more knowledge requires more knobs, and more knobs mean more work per word, forever. Picture a factory where making one item means switching on every machine on the floor — including the ones irrelevant to that item. Want more capability? Add machines. But now they all run, for everything, and the electricity bill climbs in lockstep with the capability. There is no way to grow what the factory can do without growing what it costs to run.
A mixture-of-experts model cuts that weld. Instead of one monolithic machine, it keeps a whole bank of smaller specialists — the experts — and adds a tiny decision-maker — the router — that glances at each incoming word and sends it to just a couple of the most relevant experts. The rest stay switched off for that word. So the model can store an enormous amount of knowledge across all its experts while doing only a small amount of work on any single word. The picture to hold is a large teaching hospital: hundreds of specialist doctors on staff, but each patient is triaged to just the two who fit their case and then goes home. The hospital’s total expertise is vast; no one patient consumes it all.
Two sizes that get confused
The moment a model uses only part of itself, “model size” splits into two numbers, and keeping them apart is essential. The total parameter count is everything the model stores — every expert, used or not. This is its capacity, the breadth of what it can know; it is the size of the hospital’s whole staff. The activated parameter count is the much smaller slice actually put to work on one word — the router plus the two-or-so experts it picked. This is what sets the speed and running cost; it is the handful of doctors a single patient actually sees.
Real figures make the gap vivid. Mixtral stores roughly 47 billion parameters but activates only about 13 billion per word; DeepSeek’s large model stores around 671 billion but works with only about 37 billion at a time. The whole art is buying a large total — more knowledge, better answers — while keeping the activated amount small. A slogan to carry forward: cheap to compute, expensive to store.
Figure 1. Total parameters against activated parameters.
A pinch of math: how the router chooses
Inside, an expert is nothing exotic — it is just a copy of the ordinary per-word processing stage that already lives in every block, and a mixture-of-experts model simply keeps many such copies, each with its own learned settings. The interesting part is the router, and it is worth a little precision.
For a given word, the router takes that word’s current representation $x$ and produces a raw score for each of the $N$ experts — one number saying “how much does this word want that expert?” It does this with a small learned grid $W_g$:
$$s = W_g \, x \qquad \text{(one score } s_1, \ldots, s_N \text{, one per expert)}$$Those scores are turned into proper weights with a softmax , so they are positive and sum to one:
$$g_i = \frac{e^{s_i}}{\sum_j e^{s_j}}$$Now comes the move that saves all the computation: instead of using every expert weighted by $g_i$, the router keeps only the few highest weights — the top-$k$, where $k$ is a small number like one or two — and ignores the rest entirely. Only those chosen experts actually run. Their outputs are then blended back together, each weighted by its gate value:
$$y = \sum_{i \,\in\, \text{top-}k} g_i \, E_i(x)$$Here $E_i(x)$ is what expert $i$ computes for the word. Everything not in the top-$k$ contributes nothing, because it was never consulted. The triage nurse scores every speciality and picks the best two; the other departments are not consulted. That single “keep only the top few” step is where a model with a hundred experts still only pays for two.
Figure 2. Routing one word through a bank of experts.
The deep difficulty: learning from a choice
Here is the central headache, and it explains most of what makes these models tricky. A model learns by nudging its knobs in whatever direction reduces its mistakes — but the router’s action is a choice, and a choice is all-or-nothing. You cannot send a word slightly to expert three. So the smooth learning signal has nothing to grab onto at the moment of selection. The router can be told that the experts it picked worked out and to lean into them, or that they worked out badly and to trust them less, but it never gets a clean signal saying “you should have picked a completely different expert” — because the expert it did not pick produced nothing to compare against. The router learns to weight its winners; it gets no clean nudge to change its winners.
This blind spot breeds a failure called expert collapse. Suppose, by luck at the start, a few experts are slightly better. The router sends them more words; more words means more practice, so they improve further; so the router sends them yet more. Meanwhile the neglected experts get almost no words, never improve, and fall further out of favour. The end state is a handful of experts doing nearly all the work while the rest sit idle — the storage bill for a hundred experts, the capacity of four. In the hospital, every patient goes to the same two doctors and the rest of the staff is paid to be present.
Load balancing: gently insisting the work be spread
Crucially, the ordinary “get the answer right” goal does nothing to fix this, because a collapsed model can still give reasonable answers using its few good experts. Balance has to be encouraged separately and deliberately. The fix is to add a second, mild goal alongside correctness: use all the experts roughly evenly. Concretely, we add a small load-balancing penalty to the loss — a term that grows whenever the router is overloading some experts and starving others. If $f_i$ is the fraction of words sent to expert $i$ and $p_i$ is the average gate weight it received, a common form is:
$$L_{\text{balance}} = \lambda \, N \sum_{i} f_i \, p_i$$You do not need to trace the algebra; the intuition is what counts. This term is smallest when the load is spread evenly across all $N$ experts and grows when it piles onto a few, so minimizing it nudges the router to ease off its favourites and give the neglected experts a turn. The dial $\lambda$ sets how hard this nagging pushes — too gentle and the model collapses anyway, too aggressive and the router is forced to make bad choices just to stay balanced. Tuning this pressure is the single biggest way mixture-of-experts training differs from ordinary training; a dense model has nothing to balance, because it makes no choices.
Figure 3. Collapse, and the penalty that pushes back.
Two practical headaches
Real hardware likes fixed, predictable batch sizes, so each expert is given a fixed number of slots per batch — its capacity. But routing is unpredictable, so some experts get swamped and others sit half-empty. If an expert is assigned more words than it has slots, the surplus words are simply dropped — they skip the expert and pass through unchanged, carried along by the rest of the network. A dial sets how much headroom to leave: too little and patients are turned away on busy days, too much and chairs sit empty.
The second headache is communication. Because a mixture-of-experts model stores so many experts, they do not fit on one machine and are spread across many. But the words being processed are spread across machines too, and the expert a word needs usually lives on a different machine than the word does. So before the experts run, every machine ships its words out to wherever the right experts are, and ships the results back afterward — every machine talking to every other, twice, per layer. Imagine the hospital split across a dozen buildings, with patients constantly shuttled to the right building and back, until the shuttling, not the doctoring, becomes the slow part. This heavy communication is the defining cost of training these models, and it is why balance matters twice over: a layer finishes only as fast as its busiest expert, so an imbalanced router leaves expensive machines idling at the shuttle stop.
A reality check, and the bargain that makes it worth it
It is tempting to imagine a tidy “French expert” and a “math expert.” Reality is less tidy than that. Specialization is real but weak and low-level — experts tend to react to surface patterns like punctuation and word types more than to clean human topics, and responsibility for any subject is usually smeared across several experts. The gains come less from crisp specialists than from sheer extra capacity and the router’s freedom to mix a different small combination of experts for every word.
But the gains are real, and that is the whole justification for the complexity. At a fixed amount of computing work per word, adding more experts reliably improves the model. In other words, mixture-of-experts converts a storage budget into quality without spending more compute budget. A dense model has one lever — spend more compute, get more of everything, pay for all of it on every word. A mixture-of-experts model has two levers, buying capacity with comparatively cheap storage instead of expensive computation. There is a sweet spot, since the returns shrink and the storage and shuttling are genuine costs, but that two-lever bargain is why the whole enterprise is worthwhile.
What to carry forward
A mixture-of-experts model breaks the weld between how much a model knows and how much work it does per word. It replaces the single processing stage with a bank of experts and a tiny router that scores them all, keeps only the top few, and blends their outputs — so total capacity can be enormous while the per-word cost stays small. The price of this cleverness is a cluster of problems rooted in the router’s all-or-nothing choices: expert collapse, which forces a deliberate load-balancing penalty; dropped words when experts overflow; and heavy machine-to-machine shuttling. And the defining trade in the end is memory for quality — cheap to compute, expensive to store. What the bargain does not buy is the tidy division of labour the name suggests: specialization is real, weak and low-level, with responsibility for any subject smeared across several experts, so the gain is extra capacity and the router’s freedom to mix rather than a bank of specialists.
The figures in this article follow the shared figure key , which explains what every shape, line style and colour means.
