Appendix A — Acronyms & Abbreviations

A glossary for the 10-lesson course. Each entry gives the expansion, a plain-language explanation, and where in the course it first matters. Grouped by theme so related terms sit together; use your viewer’s find (Ctrl/Cmd-F) to jump to a specific term.

This is a lookup table, not a chapter — keep it open beside the lessons and jump in when an unfamiliar acronym appears. The groups below trace the same arc the course does: the model’s components (Lesson 1–5), then the machinery that trains and aligns it (Lessons 6–9), then how it is run (Lesson 10). The symbol table at the end collects the single letters that recur as standard notation throughout.

← Course index · Appendix B — Formulas

Architecture & model components

LLM — Large Language Model. A neural network trained to predict the next token in a sequence; the whole subject of the course. (Lessons 1, 2, 7, 8)

FFN — Feed-Forward Network. The position-wise two-layer network inside each transformer block; applied independently to every token. Often used interchangeably with MLP in this context. (Lessons 1, 4)

MLP — Multi-Layer Perceptron. A stack of fully-connected layers with a nonlinearity between them; in a transformer it is the FFN. (Lessons 1, 4, 7, 9)

MHA — Multi-Head Attention. Running several attention “heads” in parallel, each on its own subspace, then concatenating and projecting the results so the model can attend to different patterns at once. (Lesson 3)

GQA — Grouped-Query Attention. An inference-time saving where several query heads share one set of key/value heads, shrinking the KV cache with little quality loss. (Lessons 1, 10)

MQA — Multi-Query Attention. The extreme of GQA: all query heads share a single key/value head, minimizing KV-cache memory. (Lessons 1, 10)

KV cache — Key-Value Cache. A buffer that stores the key and value vectors already computed for past tokens, so generation doesn’t recompute them at every step. (Lessons 1, 10)

Normalization & activations

LayerNorm — Layer Normalization. Standardizes a token’s features (subtract mean, divide by standard deviation), then applies a learned scale and shift. Stabilizes training. (Lessons 1, 4)

RMSNorm — Root-Mean-Square Normalization. A cheaper LayerNorm variant that divides by the root-mean-square of the features without subtracting the mean. (Lessons 1, 4)

GELU — Gaussian Error Linear Unit. A smooth activation function, $z\cdot\Phi(z)$, where $\Phi$ is the standard-normal CDF; a softer alternative to ReLU. (Lessons 1, 4)

SwiGLU — Swish-Gated Linear Unit. A gated activation that multiplies a linear projection by a Swish/SiLU-gated one; common in modern transformer MLPs. (Lessons 1, 4)

Positions & tokenization

BPE — Byte-Pair Encoding. A subword tokenizer that repeatedly merges the most frequent adjacent symbol pair, building a vocabulary of common chunks. (Lessons 1, 2)

WordPiece. A subword tokenizer that merges pairs by likelihood gain rather than raw frequency (used by BERT-family models). (Lesson 2)

Unigram (SentencePiece). A tokenizer that starts from a large candidate vocabulary and prunes tokens by marginal likelihood. (Lesson 2)

OOV — Out-Of-Vocabulary. A token or word the model’s vocabulary cannot represent directly; subword schemes largely eliminate this. (Lesson 2)

RoPE — Rotary Position Embeddings. Encodes position by rotating query and key vectors by a position-dependent angle, so relative position falls out of the dot product. (Lessons 1, 2)

PE — Positional Encoding. Any scheme (sinusoidal, learned, or rotary) that injects token-order information into the otherwise order-blind attention mechanism. (Lesson 2)

Training, loss & information

CE — Cross-Entropy. The standard training loss: the average negative log-probability the model assigns to the true next token. (Lessons 5, 6, 9)

KL — Kullback–Leibler divergence. A measure of how far the model’s distribution is from a target distribution; cross-entropy equals entropy plus KL. (Lessons 1, 6, 9, 10)

PPL — Perplexity. The exponential of cross-entropy loss; reads as the effective number of equally-likely choices the model faces per token. (Lessons 1, 6)

nats / bits. Units of information for entropy and loss: nats use the natural log, bits use log base 2 (bits double as a compression rate). (Lesson 6)

MAP — Maximum A Posteriori. In decoding, the goal of finding the single highest-probability output sequence. (Lesson 10)

Optimization & autodiff

SGD — Stochastic Gradient Descent. Updating parameters opposite the gradient estimated from a random minibatch — the backbone of all training. (Lessons 1, 8)

Adam — Adaptive Moment Estimation. An optimizer that keeps running averages of the gradient (first moment) and its square (second moment) to set a per-parameter step size. (Lessons 1, 8, 9)

AdamW — Adam with decoupled Weight decay. Adam but with weight decay applied separately from the adaptive update; the standard choice for training transformers. (Lessons 1, 8)

VJP — Vector-Jacobian Product. The core operation of reverse-mode autodiff (backpropagation): it pushes a gradient backward through one layer without ever forming the full Jacobian. (Lessons 1, 7)

FLOP — Floating-Point Operation. One floating-point multiply or add; the unit used to measure compute cost (e.g. the $6PD$ training estimate). (Lesson 1)

fp32 — 32-bit floating point. Single-precision numbers (4 bytes each); contrasted with lower-precision formats used to save memory at inference. (Lesson 1)

Fine-tuning & alignment

SFT — Supervised Fine-Tuning. Continuing training on curated (prompt, response) demonstrations to shape the model’s behavior. (Lessons 1, 9)

RLHF — Reinforcement Learning from Human Feedback. A two-stage post-training recipe: fit a reward model to human preferences, then optimize the policy against it. (Lessons 1, 9)

LoRA — Low-Rank Adaptation. A parameter-efficient fine-tuning method that freezes the base weights and learns a small low-rank correction $\tfrac{\alpha}{r}BA$. (Lessons 1, 9)

QLoRA — Quantized LoRA. LoRA on top of a base model quantized to 4-bit, cutting memory further so large models fine-tune on a single GPU. (Lesson 9)

PPO — Proximal Policy Optimization. A reinforcement-learning algorithm that improves the policy using a clipped objective to prevent destabilizingly large updates. (Lessons 1, 9)

DPO — Direct Preference Optimization. An alternative to RLHF that optimizes directly on preference pairs, skipping the separate reward model and RL loop. (Lessons 1, 9)

RM — Reward Model. A model trained to score outputs by predicted human preference; the target that PPO optimizes against. (Lesson 9)

Named methods & concepts

Bradley–Terry model. The preference model behind reward training: the probability one response beats another is $\sigma$ of their reward difference. (Lessons 1, 9)

Chinchilla scaling laws. Empirical finding that compute-optimal training scales parameters and training tokens together in roughly equal proportion. (Lesson 8)

FlashAttention. An IO-aware attention algorithm that computes attention in tiles, avoiding materializing the full $n\times n$ score matrix. (Lesson 3)

Goodhart’s Law. “When a measure becomes a target, it ceases to be a good measure” — invoked for reward over-optimization in RLHF. (Lesson 9)

Common single-letter symbols

These aren’t acronyms but recur as standard notation across the lessons.

← Course index · Appendix B — Formulas → · Appendix C — Attention Variants →