Appendix C — Acronyms & Symbols
Technical course on encoder LLM mechanics — reference appendix.
Where this fits
A lookup table to keep open alongside the lessons rather than read front to back. It expands every acronym and explains every recurring symbol in plain language, with the lesson where each is introduced.
Acronyms
| Acronym | Expansion | Plain meaning | Lesson |
|---|---|---|---|
| LLM | Large Language Model | a large neural network over text | 1 |
| BERT | Bidirectional Encoder Representations from Transformers | the original encoder model | 1 |
| MLM | Masked Language Modeling | predict hidden tokens from both sides | 5 |
| NSP | Next-Sentence Prediction | is sentence B the real continuation of A? (dropped) | 5 |
| SOP | Sentence-Order Prediction | are these two sentences in the right order? (ALBERT) | 5, A |
| RTD | Replaced-Token Detection | is each token original or a fake? (ELECTRA) | 8, A |
| CLS | “classification” token | the position-0 read-out register | 2, 4 |
| SEP | “separator” token | marks segment boundaries | 2 |
| AR | Autoregressive | the decoder’s left-to-right factorization | 1, 5 |
| BPE | Byte-Pair Encoding | the decoder’s usual subword tokenizer | 2 |
| WordPiece | — | BERT’s likelihood-based subword tokenizer | 2 |
| LayerNorm / LN | Layer Normalization | per-vector normalization in each block | 4 |
| post-LN / pre-LN | — | LayerNorm after vs. inside the residual | 4 |
| MLP / FFN | Multi-Layer Perceptron / Feed-Forward Network | the per-token expansion sub-layer | 4 |
| GELU / GeGLU | Gaussian Error Linear Unit / Gated GELU | block nonlinearities | 4, A |
| RoPE | Rotary Position Embedding | relative positions inside the dot product | 2, A |
| KV cache | Key/Value cache | decoder-only generation cache (encoders lack it) | 1, 3 |
| PPL / PPPL | Perplexity / Pseudo-Perplexity | decoder metric / encoder stand-in | 7 |
| GLUE / SuperGLUE | General Language Understanding Evaluation | the standard understanding benchmark suites | 7 |
| SQuAD | Stanford Question Answering Dataset | extractive-QA benchmark | 7 |
| NER / POS | Named-Entity Recognition / Part-of-Speech | token-labeling tasks | 6, 9 |
| NLI | Natural Language Inference | entailment/contradiction/neutral | 7, 9 |
| STS / MTEB | Semantic Textual Similarity / Massive Text Embedding Benchmark | embedding-quality benchmarks | 7 |
| SBERT | Sentence-BERT | bi-encoder for sentence embeddings | 10 |
| RAG | Retrieval-Augmented Generation | encoder retrieves, decoder generates | 10 |
| SFT / RLHF / DPO | Supervised Fine-Tuning / Reinforcement Learning from Human Feedback / Direct Preference Optimization | decoder alignment methods | 9 |
| LoRA | Low-Rank Adaptation | parameter-efficient fine-tuning | 9 |
| T5 / BART | Text-to-Text Transfer Transformer / Bidirectional and Auto-Regressive Transformers | encoder–decoder models | 10 |
| AdamW | Adam with decoupled Weight decay | the standard optimizer | 8 |
Symbols
| Symbol | Meaning | Lesson |
|---|---|---|
| $g(x;\theta)$ | the encoder function: tokens → contextual vectors | 1 |
| $f(x;\theta)$ | the decoder function: tokens → next-token distribution | 1 |
| $\theta$ | the learned parameter vector ($P$ of them) | 1 |
| $x=(x_1,\dots,x_n)$ | input token sequence | 1 |
| $\tilde x$ | the corrupted (masked) sequence | 5 |
| $\mathcal M$ | set of masked positions ($\approx 0.15\,n$) | 5 |
| $H^{(\ell)}\in\mathbb R^{n\times d}$ | the residual stream / hidden states at layer $\ell$ | 2, 4 |
| $H^{(N)}$ | top-layer contextual representations (the output) | 4 |
| $E_{\text{tok}},E_{\text{seg}},E_{\text{pos}}$ | token, segment, position embedding matrices | 2 |
| $W_Q,W_K,W_V,W_O$ | attention projection matrices | 3 |
| $A\in\mathbb R^{n\times n}$ | the attention matrix (full for encoders) | 3 |
| $M$ | attention mask ($0$ everywhere for encoder self-attention; $-\infty$ for padding/causal) | 3 |
| $W_U$ | output (unembedding) projection, often tied to $E_{\text{tok}}$ | 6 |
| $\mathbf c$ | pooled [CLS] (or mean) vector | 6 |
| $V$ | vocabulary size ($30{,}522$ for BERT) | 2 |
| $d$ | model width ($768$ base, $1024$ large) | 2, 4 |
| $N$ | number of layers ($12$ base, $24$ large) | 4 |
| $n,\ n_{\max}$ | sequence length; max length ($512$) | 2, 3 |
| $\Delta^{V-1}$ | probability simplex over the vocabulary | 1 |
| $\mathcal L_{\text{MLM}}$ | masked-language-modeling loss | 5 |
← Course index · reference appendix
Appendices: A — Encoder Family · B — Encoder vs. Decoder Cheat Sheet · C — Acronyms