There are things a language model is bad at that a pocket calculator from 1975 does perfectly. Multiplying two ten-digit numbers. Telling you today’s date. Reporting what a particular file contains. These are not hard problems; they are problems of a kind the model is architecturally unsuited to, because it has no persistent state, no access to the world, and no mechanism for exact computation — only a very good sense of what text usually follows what. Tool use is the arrangement that lets it stop trying. The model writes down what it wants done, something else does it, and the result comes back as text the model can read. What makes this more than plumbing is that the interface between a probabilistic text generator and a piece of software that expects exact arguments is a genuinely awkward joint, and most of the engineering is about that joint. This piece is about how the call is made, how it is made reliably, and what the model still has to get right.
What the model can and cannot do unaided
It is worth being precise about which failures a tool fixes, because the boundary is not where people expect.
The model cannot know anything that happened after its training ended, and it cannot know anything private. Both are absolute limits, not weaknesses to be improved by scale.
It cannot compute reliably. It can approximate arithmetic and often gets small cases right, but its mechanism for doing so is pattern completion over digit sequences, degraded further by the way numbers are tokenized , and it has no way to verify its own answer. Exactness is not something a next-token predictor provides.
It cannot act. Sending a message, writing a file, and placing an order are not things that happen by producing text about them.
What it can do — and this is why the arrangement works at all — is recognize which situation it is in and produce well-formed text describing what should happen. That is a language task, and language tasks are what it is for. The division of labour is therefore: the model decides what to do and with what arguments; the tool does it; the model interprets the result. The model is a router and an interpreter, not an executor.
How a call is made
Mechanically, a tool call is text the model emits in an agreed format, which the surrounding program intercepts rather than showing to the user.
The tools are described in the context — each with a name, a description of when to use it, and a specification of its parameters. The model, when it decides one applies, emits a structured block naming the tool and supplying arguments. The program parses it, executes the corresponding function, and appends the result to the conversation as a new message. The model then continues, now with the result in its context, and either answers or calls something else.
Two properties of this arrangement matter more than the format details.
The tool descriptions are part of the prompt, which means they are prompt engineering. A tool that is described vaguely will be called in the wrong situations, and a tool whose parameters are named unhelpfully will be called with bad arguments. The quality of the descriptions is a substantial fraction of the quality of the system, and the most common fix for a model using tools badly is to rewrite the descriptions rather than to change anything about the model.
The result re-enters as context, which means everything from the article on long context applies. A tool that returns ten thousand tokens of output has consumed ten thousand tokens of the model’s attention and cache, mostly with material that is not relevant. Tools should return what is needed and not what is available, and this discipline is one of the clearer dividing lines between systems that work and systems that do not.
Figure 1. The round trip of a single call.
Making the output parseable
The model must emit something a parser can consume, and a model that is merely asked to produce valid JSON will occasionally not. Three approaches address this, in increasing order of strength.
Training. Models are fine-tuned on large numbers of tool-calling examples, so that the format is something they produce naturally rather than something they are following instructions about. This raises reliability a great deal and does not guarantee anything.
Constrained decoding, which is the guarantee. As described in the article on decoding , setting a token’s logit to negative infinity makes it strictly impossible. Maintain a parser alongside generation that knows which tokens could legally continue the output given a grammar, mask everything else, and the model cannot emit malformed output — not because it has been persuaded but because the alternatives were removed from the ballot. This is the mechanism behind guaranteed-JSON features, and it costs almost nothing at generation time.
It is worth noting the honest cost, which is easy to miss. Constraining the output changes the distribution the model is sampling from, and if the model wanted to say something the grammar forbids, forcing it into the grammar may produce a worse answer rather than a well-formed good one. A model that has been constrained into valid JSON has produced valid JSON; whether the values are right is a separate question that the constraint says nothing about.
Validation and retry. Whatever the generation guarantees, the arguments still have to make semantic sense — a date that exists, an identifier that is real, a value in range. The standard arrangement validates, and on failure returns the error to the model as a tool result so it can correct itself. Models are quite good at repairing their own malformed calls when shown the error, which makes this cheap and effective.
Figure 2. Three layers of guarantee, and what each misses.
What is hard about it
Given a well-described tool and a guaranteed format, the remaining difficulties are the model’s own judgement, and they are the interesting part.
Knowing when. The failure runs both ways. Under-calling: answering from memory a question that needed a lookup, which produces a confident, plausible, stale answer — the worst failure mode of the lot, because nothing about the output signals that a tool should have been used. Over-calling: reaching for a search on something it plainly knows, wasting a round trip and context. Both are trained behaviours and both are sensitive to how the tools are described.
Choosing which. With five tools the choice is easy. With two hundred it is not, and accuracy degrades noticeably as the number grows, partly because the descriptions crowd the context and partly because similar tools become genuinely hard to distinguish. Systems with large tool sets increasingly retrieve a relevant subset first and present only those, which is RAG applied to the tool list.
Composing. Real tasks need several calls where the second depends on the first — look up the customer, then fetch their orders, then compute a total. The model must sequence these, carry intermediate values across, and recover when a step fails. This is where tool use shades into agents and where error compounding becomes the dominant concern.
Parallelism. When several calls are independent, issuing them together is much faster than one at a time. Models are trained to emit multiple calls in a single turn for this reason, and the hard part is the judgement of independence — issuing two calls in parallel when the second needed the first’s result produces a confidently wrong answer.
Trust. A tool result enters the context as text, and the model treats its context as true. If a tool returns content from anywhere untrusted — a fetched web page, a user-supplied document, an email — that content is now in a position to issue instructions the model may follow. This is prompt injection, and the article on safety and refusal argues the general case: not a bug to be patched but a structural consequence of instructions and data sharing one channel. The mitigations are architectural: mark untrusted content clearly, restrict what tools can be called after untrusted content has entered, and require confirmation for anything irreversible. There is no known way to make a model reliably ignore instructions embedded in text it has been asked to read.
Figure 3. Under-calling and over-calling.
What it changes
The broader consequence is a shift in what a model is for. If exact computation, current information, and action on the world are all delegated, then the model’s job is reduced to the thing it is uniquely good at: understanding an underspecified request, deciding what would resolve it, and interpreting what comes back.
That reframing makes some long-standing complaints look misplaced. A model that cannot multiply large numbers is not much of a problem if it reliably recognizes that a multiplication is required and calls something that can. A model whose knowledge is eighteen months stale is not much of a problem if it knows to search. What remains genuinely load-bearing is judgement — knowing what is needed, and knowing when what came back is wrong — and that is exactly the part no tool supplies.
What to carry forward
Tool use delegates what a language model is structurally unable to do — know anything after its training or outside its context, compute exactly, or act on the world — to software that can, while the model does the part it is actually suited to: recognizing the situation, deciding what should happen, and interpreting the result. Mechanically it emits a structured block naming a tool and its arguments, which the surrounding program intercepts, executes, and appends to the conversation as text. Two consequences dominate the engineering: the tool descriptions are part of the prompt, so a model calling tools badly is usually a documentation problem rather than a model problem; and every result consumes context and cache, so tools should return what is needed rather than what is available. Well-formed output is guaranteed not by asking but by constrained decoding, masking every token that a grammar forbids — which makes malformed output impossible while saying nothing about whether the values are correct, and which does change the distribution being sampled. What remains hard is the model’s judgement: knowing when a tool is needed, where under-calling produces a confident stale answer that carries no signal of being wrong; choosing among many tools, which degrades as the list grows and increasingly requires retrieving a relevant subset first; composing dependent calls in sequence and recognizing which are safe to issue in parallel; and the unresolved problem that a tool result enters the context as trusted text, so any untrusted content it carries is positioned to issue instructions the model may follow.
The figures in this article follow the shared figure key , which explains what every shape, line style and colour means.
