A team builds an agent that works beautifully for six steps and then begins repeating itself, and the diagnosis they reach for is that the model is not clever enough. They try a larger one. The repetition continues, because the problem was never the reasoning: by step twelve the original instruction sits forty thousand tokens back, sharing a window with three stale directory listings and the full text of a file the agent read once for a reason it can no longer recover. The name that has settled on the business of managing this is context engineering, which sounds like prompt writing with a grander vocabulary and is not quite that. What distinguishes it is that the decisions have measurable costs and trade against one another, which is what makes something an engineering problem rather than a set of habits. This piece is about the budget every component competes for, the ordering that the cache and the attention pattern argue about separately, and the losses that compaction and tool output quietly impose.
The window is a budget, and it is shared
Three facts, none of them remarkable alone, combine into a constraint worth stating carefully.
The window is fixed. Whatever the advertised number, it is a hard ceiling measured in tokens, and it cannot be negotiated upward at request time by any amount of care in how the request is phrased.
Everything drawing on it draws on the same allowance. The system instruction, the tool definitions, the retrieved documents, the conversation so far and the thing the user has just asked all occupy the same space, and nothing distinguishes them at the level the limit applies to.
Attention inside the window is not uniform. A token that is present is not thereby a token that is used, since the model distributes a fixed total of attention weight across every position available and material that has to compete against a hundred thousand rivals fares worse than the same material competing against a thousand.
Any one of these would be a nuisance to work around. Together they mean the interesting decisions are allocations rather than improvements, and an allocation cannot be optimized without knowing what it is being taken from. This is also why the problem does not dissolve when windows grow. A larger budget changes the numbers and leaves the structure intact, and the third fact gets worse rather than better as the first relaxes.
Figure 1. Five components, one allowance.
Order gets decided twice, by the cache and by attention
A working context for anything non-trivial holds five kinds of material at once, and every token given to one is a token unavailable to the rest. The system instruction establishes the behaviour wanted. The tool definitions describe what can be called, and with fifty tools they are not a footnote but a substantial block of prose that is re-sent every time. The retrieved documents supply the facts. The history carries what has happened. The current request is the only part that cannot be trimmed, since without it there is nothing to answer.
The cost is not only the window. It is also the money and the wait, because the whole context is read again on every turn — the model has no memory between calls, so a conversation is not continued so much as replayed with one more line. What makes the replaying affordable is the KV cache , which stores the per-token intermediate results so that a prefix already processed does not have to be processed twice. The consequence for ordering is direct and slightly unglamorous. A prefix that is byte-identical across turns can be reused; an edit anywhere invalidates the cache from that point onward, because every later token’s representation depended on what came before it. Inserting the current date at the top of a system instruction throws away the cached form of everything beneath it.
That is the strongest practical argument for putting the invariant material first — instruction, then tools, then the slowly changing material, then the volatile — and it is worth being clear that it is a systems argument about cache reuse rather than a claim about where the model reads best. The two arguments happen to agree here. They do not always.
Position matters, and unevenly. Models attend less reliably to the middle of a long input than to either end, a pattern consistent enough across long contexts to have acquired the name lost in the middle. For ordering retrieved passages this has an awkward implication: sorting passages by relevance and pasting them in order puts the fourth-best passage in the worst seat, so the strongest material belongs at the two ends with the weaker candidates buried between them. The honest caveat is that the effect varies considerably by model and has narrowed as training on long inputs has improved. It is a consideration to test, not a law to design around.
What accumulates has to be cut down
A history that outgrows the window has to be shortened, and the standard response is compaction: summarize the older turns, replace them with the summary, carry on. It works, in the sense that the conversation continues and the recent material is intact. The interesting part is the cost.
Summarization is lossy, which is unsurprising, and the loss is not uniform, which is the part that bites. What a summarizer discards is whatever looked unimportant at the moment of summarizing — and in a long task the detail that turns out to matter is frequently the one that looked incidental early, the offhand constraint in the second message, the error message that was worked around rather than fixed, the field name that appeared once. A summary that reads as a faithful account of the conversation can be missing precisely the thing needed at step forty. Every compaction is a bet that the model’s present judgement of relevance will hold for the rest of the task, and the bet is settled much later, silently, in the form of work redone.
Tool results are the usual culprit in an agent loop, more than the conversation itself. A single call can return ten thousand tokens of which four matter, and unlike a prompt the model wrote none of it and had no way to anticipate its size before asking. Three responses are available and all are crude: truncate, which risks cutting the relevant line; filter or project, returning only the fields the caller plausibly needs; or return a handle — write the payload somewhere and pass back an identifier and a summary, letting the model fetch more if it decides it needs to. This is where most of the practical work in these systems actually goes, and it is unglamorous enough that it tends to be described as plumbing rather than design.
Figure 2. Three crude answers to a ten-thousand-token tool result.
A pinch of math: the budget and the bill
Write the window as $W$ and the components as $c_1 \ldots c_n$ — instruction, tools, retrieved documents, history, request. The constraint is an inequality, not an objective:
$$\sum_{i} c_i \le W$$with one term effectively fixed, since the current request is not optional. Everything called context engineering is a choice about how the remaining slack is divided, and no allocation improves one term without taking from another.
The second piece of arithmetic concerns the bill. Let $P$ be the invariant prefix, re-sent on every turn, and suppose each turn adds roughly $R$ tokens of request and response. At turn $t$ the model reads about $P + tR$ tokens, so over $T$ turns the total read is
$$\sum_{t=1}^{T} (P + tR) \;=\; TP + R\,\frac{T(T+1)}{2}$$which is quadratic in the number of turns. A conversation at constant response length gets steadily more expensive per turn, not by any inefficiency but because the history is re-read each time it grows. Caching the prefix removes the $TP$ term and leaves the quadratic. That quadratic is the thing worth carrying away, and it is close to the only firm result on offer here.
Figure 3. The bill is quadratic in turns.
Instructions and data arrive on one channel
The model receives one undifferentiated stream of tokens. There is no privileged register for instructions and no flag marking a passage as reference material, so text that arrived as a retrieved document, a fetched web page or a tool result is available to be read as an instruction, and sometimes is. Prompt injection follows from this directly. It is not a defect of a particular model but a consequence of the architecture, and the defences — marking untrusted regions, restricting which tools may be called once untrusted content has entered, requiring confirmation before anything irreversible — are mitigations that raise the cost of an attack rather than fixes that remove it. No known method makes a model reliably ignore instructions embedded in text it has been asked to read.
The same honesty is owed to the practice as a whole, which has almost no theory in it. What is known is largely empirical, tied to particular models, and liable to go stale when the model changes; a prompt layout carefully tuned on one system frequently transfers poorly to the next, and the reasons are rarely legible. The field also disagrees with itself more than the confident tone of most advice suggests, because measuring whether a context change helped requires the same expensive evaluation machinery as any other claim about model behaviour — a task suite, repeated runs, cost accounted alongside quality — and in its absence a change that felt better on four examples becomes received wisdom. A good deal of what circulates as context engineering is at that level of evidence.
What to carry forward
Context engineering is allocation under three simultaneous facts: the window is a fixed ceiling, every component draws on that same ceiling, and attention within it is uneven, so presence in the context does not guarantee use. Those three make it an engineering problem with real trade-offs, and they are why a larger window relaxes the arithmetic without changing the structure. A working context holds a system instruction, tool definitions, retrieved documents, a history and the current request, and the cost is paid not once but on every turn, since the model has no memory between calls and the whole context is read again — the KV cache is what makes the re-reading affordable, and it rewards a stable prefix while punishing an edit near the beginning, which invalidates everything after it. That is the strongest argument for putting invariant material first, a systems argument rather than a claim about attention ; the separate observation that models attend least reliably to the middle of a long input argues for placing the strongest passages at both ends, though the effect is model-specific and has narrowed. When history outgrows the window it is compacted, and compaction is a bet that what looks unimportant now will stay unimportant, which in long tasks it often does not. Tool output is the larger practical problem, since one call can return ten thousand tokens of which four matter and the model neither wrote it nor knew its size, which is why truncation, projection and returning handles rather than payloads absorb most of the real effort. Underneath all of it sits the fact that instructions and data share one channel, so injection is structural rather than a bug, the defences are mitigations, and almost everything else here is empirical, model-specific, and held more firmly than it is known.
The figures in this article follow the shared figure key , which explains what every shape, line style and colour means.
