Chapter 5 — The Readers: Encoder Models

The storyteller of the last chapter is built to write. But writing is not the only thing one might want from a language machine. Often you already have the text — a review, an email, a legal clause, a search query — and what you want is not more text but understanding: is this review positive or negative, is this email spam, which words here are people’s names, does this query mean the same thing as that document. For jobs like these, the machinery from Part I is arranged into a different shape entirely. These are encoder models, and the most famous of them is BERT. If the decoder is a writer, the encoder is a reader.

Readers get to look both ways

The single most important difference between a reader and a writer is what each is allowed to look at. A writer, as we saw, must work strictly left to right, blind to anything ahead, because its training game forbids peeking at the answer. A reader is under no such constraint. When an encoder processes a word, it can look in both directions at once — at the words before it and the words after it.

This freedom is not a minor tweak; it is the whole point, and it changes everything downstream. Think about how much the meaning of a word can depend on what comes later. In “we sat on the bank of the river,” the word “bank” only resolves into “riverbank” once you reach “river,” which arrives afterward. A strictly left-to-right reader would have to commit to a meaning for “bank” before seeing the word that settles it. The bidirectional reader simply waits, takes in the whole sentence, and lets every word be informed by its complete surroundings.

A homely analogy: the writer is solving a crossword strictly top to bottom, never permitted to glance at the clues below. The reader fills in the whole grid with every clue visible at once, freely using answers on the right to help with answers on the left. For the job of understanding, having every clue in view is an enormous advantage.

Figure 1. The crossword, solved two ways.

What a reader actually produces

This is the part that trips people up, so it is worth stating plainly: an encoder does not produce text. It does not answer in words. What it produces, for each word you feed it, is a rich bundle of numbers — a contextual representation, which we can think of as a meaning-fingerprint — that has been shaped by the entire sentence around that word.

So the word “bank” in the river sentence comes out with a completely different fingerprint than “bank” in “I deposited the check at the bank.” Same spelling, same starting point on the map of meaning, but utterly different fingerprints once the surrounding context has done its work. That is the encoder’s entire product: not a dictionary definition but a context-aware reading of what each word means right here, in this particular sentence. A plain dictionary gives one fixed entry for “bank”; a skilled human reader instead infers which sense you meant from the sentence. The encoder is built to behave like the skilled reader, and the meaning-fingerprints are what it hands off for someone else to use.

The marker tokens that frame every input

Encoders wrap every input in a few special marker symbols that carry no word meaning of their own but do important structural work. Two are worth knowing.

The first sits at the very front of every input — think of it as a blank summary card stapled to the front of the document. Because the reader can see everything at once, this card gradually fills up, as the model reads, with a summary of the whole text. Later, when we want a single overall verdict about the entire input — is this spam, what is the topic, is the sentiment positive — we simply read that summary card. It gives the model a dedicated place to accumulate a sense of the whole.

The second is a divider, like a staple between two stapled documents, used to separate one sentence from another. It lets an encoder handle pairs of inputs at once, which is exactly what you need for questions like “do these two sentences mean the same thing?” or “does this passage answer this question?” A third marker, the blank, appears only during training, and it leads us to the cleverest part of the whole story.

How do you train a reader? Fill in the blanks

Here is a real puzzle. A writer trains by guessing the next word, billions of times. A reader cannot play that game — it can see in both directions, so the “next word” is sitting right there in plain view, and guessing it would be cheating that teaches nothing. So how does a reader learn anything from raw text?

The answer is a fill-in-the-blank game. Take a sentence, cover up about one word in seven at random, and train the model to recover the hidden words from everything around them. “The ___ sat on the mat” — the model should produce “cat.” Crucially, the model never sees the word it is trying to guess, so its ability to look both left and right is now a genuine, honest advantage rather than a cheat. It is the language-class cloze exercise, or a game of Mad Libs run in reverse.

This is the deep reason the two model families differ at the root. Bidirectionality and next-word prediction are fundamentally incompatible: you can let a model see in all directions, or you can train it to predict the next word, but you cannot do both, because seeing everything makes next-word prediction trivial. Choosing to build a reader that sees everything forces you to train it on a fill-in-the-blank objective instead. The reading ability and the fill-in-the-blank game are two sides of one coin.

There is a clever wrinkle in the recipe. If the model only ever had to think hard at the blank symbol, it would grow lazy everywhere else — but in real use there are no blanks, just ordinary complete text. So of the words chosen to be quizzed, most are blanked out, a small share are swapped for a random wrong word, and a small share are left exactly as they are while still being quizzed. The wrong-word case teaches the model not to blindly trust the word it sees, since it might be a plant. The leave-it-alone case keeps the model paying attention even to perfectly normal words, which is the situation it will always face once deployed. The effect is a reader that stays alert across the entire text, not just at the obvious gaps.

Figure 2. Fill in the blanks, and the wrinkle that keeps the reader alert.

Turning fingerprints into answers

The encoder hands off meaning-fingerprints, but a fingerprint is not yet an answer. To get an answer you bolt on a small attachment — a head — tailored to the shape of the question. The beauty is that the same understanding, captured once in the fingerprints, can be pointed at very different tasks just by changing the little head on top.

If you want one overall label for the whole text — positive or negative, spam or not — you read the summary card and run it through a tiny classifier. If you want a label for every word — which words are names, places, dates — you run a small classifier on each word’s fingerprint in turn. If you want to point at where in a passage the answer to a question lives, you have the head mark a start word and an end word, highlighting the answer span. The reader’s outputs are clean and bounded in a way a writer’s free-form text is not: a tidy probability across a fixed set of categories, or a highlighted span guaranteed to be real text from the document. It cannot wander off and invent a category that doesn’t exist. For high-volume, sensitive jobs like moderating content or tagging billions of documents, those tidy, trustworthy outputs are both cheaper and safer than asking a storyteller to write its verdict in prose.

Figure 3. Three little heads, one set of fingerprints.

Fast, single-pass, and unable to write

A reader has none of the slow word-by-word loop of a writer. It takes in the whole input in a single pass and is immediately done — there is no generating, no appending, no sampling. This is why encoders are cheap and quick as analysis tools. It is also exactly why they fundamentally cannot produce text: there is no mechanism in them for extending an output one word at a time. If you want generation, you must attach an actual writer. A reader understands; it does not speak.

This also changes how you grade one. A writer can be measured by how unsurprised it is by the next word, on average — but that yardstick depends on the left-to-right, next-word structure a reader doesn’t have, so it simply doesn’t apply. Instead, readers are judged by what they enable: attach a head, train it briefly on a real task, and see how well it scores on standard batteries of understanding problems. A reader’s worth is measured by how good its understanding turns out to be once you put it to work.

A lesson worth remembering: BERT was undertrained

One of the most instructive episodes in this corner of the field is the story of RoBERTa. Researchers took BERT’s exact design — not one structural change — and simply trained it far better: much more data, much longer, with a fresh random set of blanks each time and other refinements. The result was a large jump in quality. The startling conclusion was that the original BERT had not been designed poorly; it had merely been undertrained. Sometimes the biggest gains come not from a cleverer architecture but from feeding and training an existing one properly. It is a useful corrective to the assumption that progress always means a new design.

Why readers still matter, and how they cooperate with writers

It is tempting to think that, in the age of chatbots that can do anything, the humble reader is obsolete. The opposite is true. Encoders are the quiet, high-throughput workhorses behind a great deal of what happens on the internet: search, content moderation, classification, tagging, deduplication. They are chosen precisely because they run in a single cheap pass and produce clean, structured, trustworthy outputs.

Their biggest modern role is turning whole sentences or documents into a single meaning-fingerprint — an embedding — so that texts with similar meaning land near each other in a kind of meaning-space. This powers semantic search: finding documents that mean the same thing as your query, not merely share its keywords. And it underlies the most important pattern in applied AI today, often called retrieval-augmented generation, in which the two families cooperate beautifully. A reader, fast and search-native, finds the handful of documents most relevant to your question; a writer then composes an answer using them. The reader retrieves; the writer generates. Each does exactly what its shape was built for. Far from competing, the reader and the writer turn out to be natural partners.

What to carry forward

An encoder is a reader. It looks in both directions at once, which makes it superb at understanding but means it can never write — and which forces its training to be a fill-in-the-blank game rather than next-word prediction. Its product is a set of context-aware meaning-fingerprints, turned into clean, bounded answers by small task-specific heads. Readers are the cheap, fast, trustworthy workhorses behind search, classification, and the retrieval half of modern AI systems.

We have now met the writer and the reader. The obvious next question is: what if you need both — to read one thing thoroughly and then write a different thing based on it, as in translation or summarization? That calls for bolting a reader and a writer together, which is the subject of Chapter 6 .

← Chapter 4 · The Storytellers · Contents · Chapter 6 · The Translators →

The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.