Chapter 2 — From Words to Numbers

In the last chapter we settled what a model is: a function, set by billions of learned knobs, that turns an input into an output. But there is a gap in that picture we glossed over. A computer, at bottom, only ever does arithmetic. It manipulates numbers. And yet we hand it words — fuzzy, ambiguous, deeply human things. Something has to bridge that gap before any of the clever machinery can begin. This chapter is about that bridge. It comes in two steps: first chopping text into pieces, and then turning each piece into a point on a vast “map of meaning.”

Step one: chopping text into tokens

The first thing that happens to your text, before the model does anything intelligent with it, is that it gets cut into chunks. These chunks are called tokens. You might expect a token to be a word, but it usually isn’t quite. Tokens tend to be common word-fragments. The word “tokenization” might be split into “token” and “ization.” Frequent short words like “the” stay whole; rarer or longer words get broken into familiar pieces.

Why pieces rather than whole words? It is a practical compromise between two bad extremes. If you insisted on whole words only, the model would be helpless the first time it met a word that wasn’t on its list — a new name, a typo, a piece of slang, a technical term. There are endlessly many possible words, and no fixed list can hold them all. But if you went to the opposite extreme and fed the model individual letters, every sentence would become an enormously long string of tiny pieces, and the model would drown in length. Word-fragments thread the needle: common words stay efficient and whole, anything unusual breaks down into known fragments, and as a last resort literally anything can be spelled out letter by letter. The model is never stumped by an unfamiliar word; it just assembles it from smaller parts.

This chopping is done by a fixed recipe built once, in advance. It never learns and never changes. It is a dumb, mechanical lookup — and at this stage each token is nothing but an identification number, like a coat-check ticket. The number means nothing in itself. “Token number 4,219” carries no meaning yet; it is just a stub you can redeem for the real thing in the next step.

It is worth knowing that this innocent chopping step is the source of some of the famous quirks of language models. When a model fumbles simple arithmetic, or miscounts the letters in a word, it is often because the chopping did not line up with the digits or letters in the way a human would. The model never saw the word the way you see it; it saw a handful of fragments.

Figure 1. Three ways to chop up text, and the one that shipped.

Step two: turning tokens into points on a map of meaning

Here is where it gets beautiful. Each token’s ID number is looked up in a giant table — itself one of the learned things, written during training — that converts the number into a long list of numbers. That list is the token’s embedding. The intuition you want is this: each token is being placed at a specific location in an enormous, many-dimensioned “map of meaning.”

On this map, location encodes meaning. Words that mean similar things end up near one another; words used in very different ways end up far apart. “King,” “queen,” and “prince” settle into one neighborhood. “Tuesday” and “Wednesday” cluster in another, across the map. Nobody assigns these locations by hand. The geometry emerges during training, for the simple reason that placing similar words near each other helps the model predict well — and predicting well is the only thing training rewards.

The genuinely surprising part is that directions on this map turn out to carry meaning too, not just neighborhoods. The step that takes you from “man” to “woman” is roughly the same step, in the same direction, that takes you from “king” to “queen.” Move from a country to its capital and the displacement is consistent. This is the celebrated observation that, on the map of meaning, “king minus man plus woman lands you near queen.” It is not programmed. It falls out, unbidden, from the training process — meaning quietly reorganizing itself into geometry.

This is the deeper reason embeddings matter so much. They are how raw symbols become semantic. Once a word is a location on a map where nearness means similarity and direction means relationship, all the later machinery has something rich to work with. The arithmetic the computer is so good at can now operate on meaning, because meaning has been turned into coordinates.

Figure 2. From a coat-check ticket to a location on the map of meaning.

The problem of order

There is one thing a location on the map of meaning does not capture: where the word sat in the sentence. And order plainly matters. “Dog bites man” and “man bites dog” are built from the very same words, yet they describe opposite events. If all the model had were the meaning-coordinates of the words, with no sense of their arrangement, it could not tell these apart.

So alongside each word’s meaning, the model is also given information about its position. The clever modern approach does not just stamp each word with an absolute seat number (“you are word number seven”). Instead it encodes relative distance — how far apart any two words are from each other. This turns out to be what actually matters for understanding language, and it has a happy side effect: a model built around relative distances handles longer texts more gracefully than one trained to recognize fixed seat numbers it may never have seen during training. The names attached to these schemes can get baroque, but the idea is plain: the model needs to know not only what each word means but where it stands in relation to the others.

Putting the two steps together

Let us trace a sentence through the bridge. You type a sentence. First it is chopped into tokens — mostly familiar word-fragments, each now just an ID number. Then each ID is looked up and placed at its location on the map of meaning, and stamped with information about its position in the sentence. What was a string of human characters is now a sequence of rich numerical points, each one carrying both what the word means and where it sits. The fuzzy human thing has become something the model’s arithmetic can grip.

And yet something is still missing, and it is the most important thing of all. Right now, each word’s meaning-coordinate was computed in isolation, by simple lookup. The word “bank” gets the same starting location whether the sentence is about a river or about money. The word “it” arrives with no idea what it refers to. Each word, so far, is an island. Before the model can understand anything, the words have to talk to each other — to look around, figure out which of their neighbors matter, and revise their own meaning in light of the company they keep.

That conversation among words is the mechanism at the very heart of every model in this book. It is called attention, and it is the subject of Chapter 3 .

Figure 3. The whole bridge, and the thing still missing at the end of it.

← Chapter 1 · What These Machines Are · Contents · Chapter 3 · Attention →

The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.