Articles on Ship of Theseus

Ablation — Learning What a Model Needs by Taking Pieces Away

Sun, 23 Aug 2026 00:00:00 +0000

There is an old and reliable way to find out what something does: remove it and see what breaks. Surgeons and neuroscientists have done this for centuries — lesion a small region of tissue, watch which ability fades, and infer that the region was responsible for it. The word is ablation, literally a carrying-away, and it has become one of the few tools that gets inside a large language model at all. A modern model is a black box of billions of interacting knobs, and nothing can be read off it directly. So the next best thing gets done instead. A piece is taken away — a layer, an attention head, a training trick, a single direction in the model’s internal space — the model runs again, and the damage is measured. The size of the damage is the measure of the part’s importance. This piece is about how that removal is defined, how the number is read, and how little a small number proves.

Articles on Ship of Theseus

Ablation — Learning What a Model Needs by Taking Pieces Away

AdamW — Letting Every Knob Move at Its Own Pace, and Pulling Them All Home

Agents — Turning One Answer Into a Sequence of Actions

Arithmetic Intensity — Why Moving Numbers Costs More Than Using Them

Attention — Letting Every Word Ask the Others What It Needs

Batch Size — How Much Evidence to Gather Before Each Step

Chain-of-Thought Faithfulness — Whether the Reasoning Shown Is the Reasoning Done

Context Engineering — Deciding What the Model Gets to See

Continuous Batching and Paged Attention — Keeping the Machine Fed

Cross-Entropy — Measuring Wrongness in Bits

Data Mixtures — Deciding What a Model Reads, and When

Decoding — Choosing a Word the Model Only Suggested

Diffusion Language Models — Writing a Sentence All at Once

Distillation — Teaching a Small Model to Think Like a Big One

DPO — Reading the Reward Off the Model Itself

Embeddings — The Two Maps a Model Keeps of Its Vocabulary

Emergent Abilities — When a Curve Looks Like a Cliff

Evaluation — Measuring Something That Refuses to Hold Still

Feed-Forward Layers — Where a Model Keeps What It Knows

FlashAttention — Never Writing Down the Matrix You Do Not Need

Gradient Checkpointing — Trading Recomputation for Room

Gradient Descent and Backpropagation — How a Model Learns Which Way to Move

Grouped-Query Attention — Paying for Fewer Copies of the Same Notes

Hallucination — Why a Model Cannot Tell You What It Does Not Know

Induction Heads — The First Circuit Anyone Fully Understood

Inference Quantization — Shrinking a Finished Model to Fit

Initialization and muP — Starting So That the Dials Transfer

KL Divergence — The Distance Between Two Sets of Odds

KV Caching — Reading Each Word Once, and Carrying It Forever

Learning From AI Feedback — Replacing the Annotator With the Model

Learning-Rate Schedules — Deciding When to Slow Down

Linear Attention — Trading Exactness for a Cost That Does Not Explode

Logits — The Scores Beneath the Odds

Long Context — When the Window Outgrows the Attention

LoRA — Teaching an Old Model New Tricks Without Retraining It

Loss Spikes — When a Training Run Falls Over

Memorization — What a Model Keeps Verbatim, and When That Matters

Mixed Precision — Training in Half the Bits Without Losing the Thread

Mixture of Experts — A Model That Only Switches On What It Needs

Multimodality — Teaching a Language Model to See

Normalization — Keeping the Signal Steady Enough to Stack

Parallelism — Splitting a Model Five Different Ways

Positional Encoding — Putting Order Back Into a Mechanism That Ignores It

PPO — Learning From a Judge Without Believing It Too Much

Pretraining Objectives — Why Predicting the Next Word Was Enough

QLoRA — Fine-Tuning a Giant on a Single Graphics Card

RAG — Giving a Model Something to Read Before It Answers

Reasoning Models — Teaching a Model to Think Before It Speaks

Regularization at Scale — Why the Old Defences Stopped Mattering

Retrieval Embeddings — Turning Meaning Into a Searchable Coordinate

Reward Models — Turning Human Taste Into a Number

RLVR — Letting the Answer Key Do the Teaching

RoPE — Telling a Model Where a Word Sits by Turning It

Safety and Refusal — Teaching a Model to Decline

Scaling Laws — Predicting Quality Before You Train

Softmax — Forcing a Choice When There May Be None to Make

Sparse Attention — Choosing Who Is Allowed to See Whom

Speculative Decoding — Guessing Ahead and Checking in Bulk

State Space Models — Carrying the Past in a Fixed-Size Summary

Superposition — Packing More Ideas Than There Are Neurons

Supervised Fine-Tuning — Teaching a Model to Answer Rather Than Continue

Test-Time Compute — Buying Accuracy With Sampling

The Residual Stream — The Shared Channel Every Layer Reads and Writes

The Transformer Block — The Unit That Gets Stacked

Tokenization — Cutting Language Into Pieces a Model Can Count

Tool Use — Letting a Model Call Something That Actually Knows