Appendix C — Acronyms & Symbols#
Technical course on MoE LLM mechanics — reference appendix for all lessons.
Where this fits#
A lookup table to keep open alongside the lessons. It covers the symbols and acronyms specific to MoE; for the shared transformer vocabulary (attention, $Q/K/V$, LayerNorm, AdamW, cross-entropy, perplexity, etc.) see the companion dense course’s acronym appendix. Terms below are grouped by what they describe.
Core MoE symbols#
| Symbol | Meaning | Lesson |
|---|
| $E$ | number of experts per MoE layer [CONFIG] | 1, 2 |
| $k$ | number of experts activated per token (top-$k$) [CONFIG] | 1, 3 |
| $E_i(\mathbf x)$ | expert $i$ — an FFN with its own weights [LEARNED] | 2 |
| $W_g\in\mathbb R^{E\times d}$ | router (gating) matrix [LEARNED] | 3 |
| $\mathbf h=W_g\mathbf x$ | router logits (one score per expert) [STATE] | 3 |
| $\mathbf P=\mathrm{softmax}(\mathbf h)$ | router distribution over experts [STATE] | 3 |
| $\mathcal T_k(\mathbf x)$ | set of top-$k$ selected expert indices for a token [STATE] | 3, 4 |
| $g_i(\mathbf x)$ | gate weight: renormalized prob. of selected expert $i$ [STATE] | 3, 4 |
| $\mathbf x$ | token representation entering the MoE sub-layer | 2–4 |
Parameter accounting#
| Symbol | Meaning | Lesson |
|---|
| $P_{\text{tot}}$ | total parameters — capacity, drives memory | 1 |
| $P_{\text{act}}$ | activated parameters per token — drives compute | 1 |
| $k/E$ | sparsity ratio (fraction of experts active) | 1, 4 |
| $d$ | model (residual) width | 2 |
| $d_{ff}$ | expert hidden width (typically $4d$) [CONFIG] | 2 |
| $m$ | granularity factor (fine-grained experts) [CONFIG] | 2 |
| $\mathcal S$ | set of shared (always-on) experts [CONFIG] | 2 |
Balancing#
| Symbol | Meaning | Lesson |
|---|
| $f_i$ | fraction of batch tokens dispatched to expert $i$ [STATE] | 5 |
| $P_i$ | average router mass on expert $i$ over the batch [STATE] | 5 |
| $\mathcal L_{\text{aux}}=\alpha E\sum_i f_iP_i$ | Switch auxiliary load-balancing loss | 5 |
| $\alpha$ | auxiliary-loss weight [CONFIG] | 5 |
| $\mathcal L_z=\beta\,\overline{(\log\sum_i e^{h_i})^2}$ | router z-loss (logit stability) | 5, 8 |
| $\beta$ | z-loss weight [CONFIG] | 5 |
| $b_i$ | per-expert selection bias (aux-loss-free balancing) | 5 |
| $\gamma$ | bias update rate [CONFIG] | 5 |
Capacity & communication#
| Symbol | Meaning | Lesson |
|---|
| $T$ | number of tokens in a batch | 6 |
| $C_f$ | capacity factor ($\ge 1$) [CONFIG] | 6 |
| $C=\lceil C_f kT/E\rceil$ | expert capacity (fixed buffer size per expert) | 6 |
| $D$ (tensor) | dispatch tensor — scatters tokens into expert buffers [STATE] | 6 |
| $\tilde D$ | combine tensor — gate-weighted un-scatter [STATE] | 6 |
| $G$ | expert-parallel degree (devices over which experts are sharded) [CONFIG] | 7 |
| $D$ (scalar) | number of training tokens (in $C\approx 6P_{\text{act}}D$) | 1, 8 |
Acronyms#
| Acronym | Expansion | Lesson |
|---|
| MoE | Mixture of Experts | all |
| FFN | Feed-Forward Network (the expert’s internal structure) | 2 |
| top-$k$ | selecting the $k$ highest-scoring experts | 3 |
| all-to-all | collective where every device exchanges data with every other | 7 |
| EP | Expert Parallelism | 7 |
| DP / TP / PP | Data / Tensor / Pipeline Parallelism (inherited from dense) | 7 |
| z-loss | router log-partition penalty (numerical stability) | 5, 8 |
| GShard | the framework that standardized expert parallelism | 7, A |
| ST-MoE | “Stable and Transferable MoE” (z-loss, fine-tuning) | 8, A |
| GLaM | “Generalist Language Model” (MoE decoder LM) | 8, A |
| LoRA | Low-Rank Adaptation (parameter-efficient fine-tuning) | 9 |
| SFT / RLHF / DPO | the alignment pipeline (inherited from dense) | 9 |
| KV cache | cached keys/values for fast decode (inherited, unchanged) | 10 |
| GQA / MQA | Grouped- / Multi-Query Attention (KV-cache savings, inherited) | 10 |
| $P_{\text{tot}}$ / $P_{\text{act}}$ | total / activated parameters | 1 |
The two invariants, restated#
- Sparse activation: all of $\theta$ is stored and trained, but only a $k/E$ slice of the experts is computed per token — so memory follows $P_{\text{tot}}$, compute follows $P_{\text{act}}$.
- A free router collapses: sparse routing requires an explicit balancing pressure (auxiliary loss, z-loss, or bias control) that dense training never needs.
← Course index
· Reference for all lessons
Appendices: A — MoE Family
· B — Dense vs. MoE
· C — Acronyms