Chapter 6 — The Translators: Encoder–Decoder Models

We have met the writer, who generates text from left to right, and the reader, who understands text in a single bidirectional pass. Now consider a job that seems to need both at once: translation. To translate an English sentence into German, you must first read the English completely — understanding it fully, including how its ending clarifies its beginning — and then write a fresh German sentence based on that understanding. Summarization is the same shape: read a long article thoroughly, then write a short summary. Answering a question about a passage is the same shape again. These “read one thing, write a different thing” tasks have a name — sequence-to-sequence — and they are the natural home of the encoder–decoder model, the family that includes T5 and BART.

Why these tasks want two stacks

What makes translation and summarization different from plain storytelling is an asymmetry. The input is complete and available from the very start, and it never changes. The output, by contrast, has to be built up one word at a time. You have a finished thing to read and an unfinished thing to write, and they are not the same length or even the same language. That mismatch is the whole reason a special architecture exists.

The natural response is to split the work between two specialists. An encoder — a reader, exactly the bidirectional reader from the last chapter — digests the entire input and turns it into a rich internal memory. A decoder — a writer, exactly the left-to-right storyteller from Chapter 4 — produces the output one word at a time, and at every step it consults that memory. This mirrors what a human translator actually does: read the whole source sentence carefully first, then start writing the translation, glancing back at the original as needed. You would never translate word by word having read only the opening; you read it all, then write.

Splitting the work this way buys three concrete things. The reading happens bidirectionally, so the input is understood fully. The reading happens once, and the resulting memory is reused for every output word, which is efficient. And the two roles are decoupled, so the reader can specialize in understanding while the writer specializes in producing fluent output.

The one sentence that captures the whole architecture

Here is the single most useful thing to hold in your head about this entire model family: the encoder compresses the context, the decoder generates conditioned on it, and the bridge between them — called cross-attention — is the only genuinely new mechanism. Almost everything else you would need to understand an encoder–decoder model, you already know. The map of meaning, attention, the towers of blocks, the output head, the training game — all of it is borrowed, essentially unchanged, from the writer and reader we have already met. Only one new piece is added. So if we understand that one new piece, we understand the whole thing.

Cross-attention: the only new idea

The reader produces its memory — a set of meaning-fingerprints, one per input word. The writer then generates the output one word at a time. The new question is: how does the writer, while composing each word, draw on the reader’s memory of the input? The answer is cross-attention, and it is just a small variation on the attention you already understand.

Recall the party from Chapter 3: each word poses a question, every word offers a label and some information, and matches let information flow. Ordinary attention has a sentence asking questions of itself. Cross-attention changes exactly one thing: the questions come from the writer (the half-finished output), while the labels and information come from the reader (the memory of the input). That is the entire difference — a change of who asks and who answers.

Picture the human translator again, partway through a German sentence. To choose her next word, she asks herself, “which part of the English source do I need right now?” She scans the source, her eye lands most heavily on the relevant word or two, and she pulls their meaning across to shape what she writes next. Cross-attention is exactly that scan. As the writer composes each output word, it poses a question, surveys the reader’s memory of the input, finds the input words that best answer it, and draws their meaning in. The pattern of where its attention lands — strong on this source word, weak on that one — amounts to a soft, automatic alignment between the output and the input. For translation it tends to light up right where each output word corresponds to a source word. This learned, flexible alignment is what replaced the rigid, hand-built translation tables of older systems.

Two details are worth noting. Cross-attention has no “no peeking” rule — the writer may look at any part of the input, including parts that correspond to later positions, because the whole input is already known and there is nothing to spoil. And cross-attention is the only channel through which the writer ever draws on the input. Cut it, and the writer would simply ignore the source and ramble on its own. It is the bond that makes the two stacks one machine.

Figure 1. Cross-attention: the questions change hands.

Three designs on a single spectrum

A natural objection arises here. Modern chatbots are decoder-only — pure storytellers — and yet they translate and summarize perfectly well, just by being given the input as part of their prompt and continuing from it. So do we really need two separate stacks?

The elegant answer is that all of these designs are points on a single spectrum, and the dial that distinguishes them is just one rule: what is each word allowed to look at? At one end sits the encoder-only reader, where every word sees every other word — a superb understander that cannot generate. At the other end sits the decoder-only writer, where every word sees only what came before — a superb generator that reads its input merely as more text to continue. The encoder–decoder sits in between by a different route: it gives reading and writing their own separate stacks, bonded by cross-attention. And there is an even leaner middle option, sometimes called a prefix language model, which uses a single stack but lets the input portion look around freely while keeping the output portion strictly left-to-right — getting the main benefit of bidirectional reading without a second stack at all.

Reading across that spectrum, the looking rule gets steadily stricter and the way the input gets folded in gets steadily more implicit. This is the deep reason a decoder-only chatbot can translate at all: a left-to-right stack plus a well-crafted prompt approximates the prefix model, which approximates the full encoder–decoder. The field largely consolidated on decoder-only models for general-purpose use because one uniform stack is simpler to scale up. But the dedicated two-stack design persists wherever reading the input really well is decisive — and controlled experiments have found it genuinely strongest for translation and summarization.

Figure 2. One dial: what is each word allowed to look at?.

How they learn: repairing damaged text

To train these models on mountains of unlabeled text, you need a task that manufactures its own answers. The trick both T5 and BART use is denoising: take a piece of clean text, deliberately damage it, feed the damaged version to the reader, and train the writer to reconstruct what was lost. The damage itself creates a matched input-output pair out of a single document — no human labeling required.

T5’s version is a fill-in-the-blanks puzzle on a larger scale: blank out a few short stretches of words, mark each with a placeholder, and have the writer produce just the missing pieces. Because the writer only has to generate the small missing bits rather than the entire document, training stays cheap, and because a blank’s contents depend on words on both sides, it forces genuinely bidirectional reading. BART’s version is a grab-bag of corruptions — masking words, deleting words so the model must work out where something went missing, shuffling sentence order, and more — with the writer rebuilding the whole original. The variety is the point: learning to undo many kinds of damage builds exactly the robust reading and fluent writing that translation and summarization demand.

One subtlety connects back to cross-attention. Only the writer produces the final output and therefore the only direct measure of error. So how does the reader ever improve? Entirely through cross-attention, which is the single channel carrying the learning signal back into the reader. The two stacks are trained together as one, and over time the reader learns to produce exactly the kind of memory the writer keeps reaching for. (The general mechanics of how this learning signal flows are the subject of Chapter 9 .)

Using a translator: read once, write many

When a trained encoder–decoder model is put to work, the division of labor is clean and efficient. The reader runs exactly once on the input, producing the memory. Then the writer generates the output one word at a time, each step consulting that same fixed memory. The expensive reading is paid for once and reused for every output word.

Because tasks like translation have a roughly correct answer — unlike open-ended creative writing — these models typically choose their words a little more carefully than a freewheeling chatbot. A common method keeps several promising partial sentences alive in parallel and lets the best overall one win, rather than greedily grabbing the single most likely next word and risking painting itself into a corner. There are also characteristic failure modes to manage: because the writer freely re-consults the input without remembering what it has already covered, it can fixate on one source phrase and repeat it, or skip another and drop it. Techniques that track how much of the source has been “covered” help discourage both. And for things that must be reproduced exactly — a name, a number, a rare word — the model can be given the ability to copy a word straight from the source rather than trying to regenerate it from scratch.

Figure 3. Read once, write many.

T5’s big idea, and how T5 and BART differ

T5 is built around one organizing insight: every language task can be reshaped as “text in, text out,” which is exactly the shape an encoder–decoder already has. So rather than bolting different machinery onto the model for each task, you simply describe the task in plain words at the front of the input and read the answer as text. Prefix the input with “translate English to German:” and the output is the translation. Prefix it with “summarize:” and the output is a summary. Even a yes-or-no classification becomes “generate the word yes or no,” and a numeric score becomes “generate the number as text.” One model, one setup, one method, handling everything — only the wording of the input and the training data change.

As for the two famous names: both are encoder–decoder models sharing the identical cross-attention core, and they differ mostly in their pretraining damage and minor engineering choices. BART is essentially the classic original design, taught to repair messy text through its grab-bag of corruptions, and it is especially strong at summarization. T5 keeps the same skeleton but reframes everything as text-to-text and trains with fill-in-the-blank span corruption. Think of BART as “the classic translator, taught to fix damaged text” and T5 as “the same machine, taught that every task is just text-in, text-out.” The mechanism that makes them work is the same in both — and, remarkably, the same idea reappears far beyond text, bonding a reader and a writer in systems that turn speech into transcripts or images into captions.

What to carry forward

An encoder–decoder model bolts a reader to a writer for the “read one thing, write a different thing” jobs of translation and summarization. The reader compresses the input into a memory; the writer generates the output while consulting that memory through cross-attention — the one genuinely new mechanism, which is just attention with the questions coming from the writer and the answers from the reader. All three language-model shapes — reader, writer, and the two-stack translator — sit on one spectrum defined by what each word is allowed to see.

So far, every model we have met uses all of itself to process every word. The next chapter asks a radical question: what if a model could be enormous in what it knows, yet only switch on a small part of itself for any given word? That is the idea behind mixture-of-experts, in Chapter 7 .

← Chapter 5 · The Readers · Contents · Chapter 7 · The Committee →

The figures in this lesson follow the shared figure key , which explains what every shape, line style and colour means.