Tool calling boundary
A Socratic walk-through of the tool calling boundary — reasoned out one step at a time, not lectured.
The question we started with
THE QUESTION #Why does a model that calls a tool never actually run it?
The phrase is everywhere: the model called a function. It queried the database. It sent the email. The language is so natural that the arrangement it describes disappears behind it.
But consider what a model produces. Tokens. Only ever tokens. If that is all it emits, then "calling a function" must mean something other than what it means when your code calls a function — and the difference turns out to be the single most important safety property of the whole design. So let us take the phrase apart.
Reasoning it through
REASONING #Suppose you wanted a model to fetch a customer record. What could it possibly emit that would achieve this? Not the record — it does not have one. The most it can produce is a description: a name of an operation and some arguments, written in a form the surrounding program agreed to recognise.
So the model's entire contribution is a well-formed request. Its output is a proposal. And a proposal has no effect until something else accepts it.
Now ask what that something else is and what it has that the model does not. It has the database connection. It has the credential. It has the network. And, crucially, it has a list — the tools it is willing to run at all. Anything the model asks for that is not on the list is simply text that matched nothing.
Notice what this arrangement rules out. The model cannot invent a new capability, because inventing one means writing a name the harness does not know, and the harness does nothing with an unknown name. It cannot escalate, because it never held a credential to escalate from. It cannot act without a record, because acting is the harness's doing and the harness can log it. Every one of those guarantees comes not from the model being well-behaved but from the model being unable — which is a far stronger kind of assurance.
This is an old idea in a new setting. Separating the component that decides from the component that has the authority to execute is what operating systems do with system calls: a program cannot touch a disk sector, it can only ask the kernel to, and the kernel checks. The model here sits in the position of the unprivileged program. Its persuasiveness is irrelevant to the check.
But now the harder question, and it is where the elegance of the boundary meets its limit. What comes back through it? A tool result — a document, a search snippet, a database row — which is appended to the model's context and read as ordinary input. And the model has no reliable way to distinguish "text my operator wrote" from "text that arrived in a search result". So instructions hidden in a fetched web page can influence what the model asks for next. That is prompt injection, and it is not a bug in the boundary. It is the return path working exactly as designed, carrying untrusted content into the one place that decides what to request next.
Which sharpens what the boundary actually promises. It guarantees that only permitted operations happen. It does not guarantee that the reason a permitted operation was requested is sound. And I should be plain that no general defence for this is settled; layered mitigations exist, but treating tool output as untrusted input remains the honest posture.
The analogy
THE ANALOGY #Think of a customer at a bank counter and the teller behind the glass. The customer can ask for anything — a withdrawal, a transfer, a balance. They cannot reach the drawer. The teller checks the account, the limit, the identification, and either performs the operation or does not. The customer's confidence, phrasing, or urgency changes nothing about what the teller is permitted to do.
a teller can notice that a customer seems to be acting under someone else's instruction and pause, whereas the harness checks only the form of the request against its list — so a well-formed request that was planted by a poisoned document passes exactly as cleanly as a legitimate one.
Clarifying the model
THE MODEL #Three refinements.
First, the vocabulary. "The model called the tool" compresses four distinct events: the model emitted a structured request, the harness parsed and validated it, the harness executed with its own authority, and the result was appended for the model to read. Three of those four are ordinary code, and all of the risk-bearing behaviour lives there. Naming them separately is not fussiness — you cannot secure a step you have collapsed into a metaphor.
Second, what makes tools reliable is not the model but the schema. Tool definitions specify names, argument types, and required fields, and the harness rejects a request that does not fit. A malformed request is an error the model can be shown and asked to correct, not an action with a strange effect. The tighter the schema, the smaller the space of things that can be requested at all — which is why narrow, specific tools are safer than one general "run this" tool, and why the latter dissolves the boundary almost entirely.
Third, the misconception to retire: that a model with tools has been given power. It has been given a vocabulary. The power stayed with the program, which chose which words in that vocabulary it would honour — and can change that choice without touching the model, which is precisely why permissions belong in the harness rather than in an instruction asking the model to behave.
A picture of it
THE PICTURE #How to readRead top to bottom and watch which lifeline each arrow starts from. Every arrow that touches the database starts at the harness, never at the model — that gap is the boundary. The self-directed arrow is the check that makes the arrangement worth anything, and the crossed arrow beside it is what happens to a request the harness does not recognise. The final note marks the direction the boundary does not protect: content flowing back can carry instructions, and the model reads it the same way it reads yours.
What became clearer
WHAT CLEARED #A model never runs anything. It writes a request, and a privileged program decides whether that request is one it is willing to perform. Everything an agent can do is therefore enumerated in code somewhere, which is a much better place to control behaviour than a paragraph asking the model to be careful. The boundary is genuinely strong in the outbound direction and genuinely leaky in the inbound one — what the model may do is constrained, what may influence its next request is not.
Where to go next
ONWARD #- Why narrowly scoped tools are safer than one general execution tool, and what that costs in flexibility.
- How prompt injection through tool results is currently mitigated, and why none of the approaches is complete.
- What changes when a tool's effect is irreversible, and where human confirmation belongs in the loop.
Key terms
TERMS #| Term | What it means |
|---|---|
| Tool call | a structured request emitted by a model naming an operation and its arguments. |
| Harness | the surrounding program that validates, executes, and returns results, holding all real authority. |
| Schema | the declared shape of a tool's arguments, against which a request is validated before execution. |
| Privilege separation | keeping the component that decides apart from the component permitted to act. |
| Prompt injection | instructions embedded in content the model reads, influencing what it requests next. |
Every term the collection defines is gathered in the glossary.