Thinking budget
A Socratic walk-through of a thinking budget — reasoned out one step at a time, not lectured.
The question we started with
THE QUESTION #Why can a model get a hard problem right by being given more time rather than more knowledge?
Two runs of the same model, on the same problem, with the same weights. One is told to answer; one is allowed to spend several thousand tokens working before it answers. The second is right more often.
Nothing was learned in between. Not a single parameter changed. Whatever the model knows, it knew already in the first run. So where did the improvement come from — and what does it mean to say a fixed system got better at something by being given time?
Reasoning it through
REASONING #Ask first what a model's parameters actually store. Not answers — there are more possible questions than parameters. What they store is something more like a capacity: the ability to take a step, recognise a pattern, apply a rule. Knowing a rule and having executed it on this particular input are different states of affairs, and only the first is in the weights.
So consider a problem where every individual step is within the model's competence but the path from question to answer is twelve steps long, with two places where the obvious step is the wrong one. Is this a knowledge problem? It does not look like one. Nothing is missing. What is missing is the doing — the execution of a search that the model is capable of but has not performed.
And here we can lean on what we know about the architecture: a transformer does a fixed amount of computation before each token. If the answer must come immediately, twelve steps of search have to happen in that fixed budget, which they cannot. If the model may write first, each written token buys another full pass. Extra tokens are, quite literally, extra computation.
That is the whole trick, and it deserves to be stated plainly: the resource being added is compute at inference time, not information. Which is why the phrase "more time" is the right intuition and slightly the wrong word — a model given more wall-clock time and told to answer in one token gains nothing. The budget is measured in tokens, because tokens are what turn into passes.
Now push on the shape of the return. Would you expect the second thousand tokens to help as much as the first? Consider what they get spent on. The first few hundred set up the problem and take the obvious path. Later tokens go to checking, to backtracking, to trying a second approach. Each is worth less on average than the last, because the cheap wins were taken first — and once the model is on a wrong track, more tokens can elaborate the error rather than escape it. Reported scaling curves in this area rise steeply and then flatten; I will not attach a number to the flattening point, because it moves with the model, the task, and the year.
Two more honest limits. A budget cannot supply a fact the model does not have — it will produce longer, more confident wrong working, which we saw with chain-of-thought and which does not improve with more of it. And the benefit depends on the model being able to tell whether a step went well; verification is what makes search worth running, and a model that cannot recognise its own error can only wander. That is precisely why this became a training target rather than a prompting trick: OpenAI's o1, released in preview in September 2024, and the extended-thinking modes that followed in models such as Claude 3.7 Sonnet in February 2025, are trained to use a long private working stretch productively, with an explicit budget the caller can set.
The analogy
THE ANALOGY #Think of a chess player with a clock. Their strength — the openings they know, the patterns they see — is fixed the moment they sit down. Give them thirty seconds a move and they play their first instinct. Give them thirty minutes and they calculate lines, spot the refutation, choose differently. They learned nothing at the board. They searched more of what they already knew how to evaluate, and the extra minutes bought exactly that.
the chess player's thinking is private and costs the tournament nothing, whereas a model's search is generated tokens that are paid for and take real seconds — so unlike the player, who may as well use their full clock, a model's budget is a genuine trade against cost and latency on every single call.
Clarifying the model
THE MODEL #Three refinements.
The first corrects the natural framing. "The model is thinking harder" suggests an effort dial. There is no such dial. What changes is how many forward passes get spent before an answer is committed, and the mechanism for spending them is generating text — usually hidden from the user, but text nonetheless. Effort here is a quantity of computation, not a quality of attention.
The second is where this sits relative to training. Making a model bigger or training it longer raises the ceiling of what a single pass can do. A thinking budget raises how much of that ceiling gets used on this particular question. They are complements, and they have opposite cost structures: training is paid once for everyone, thinking is paid every time anyone asks. That is why a thinking budget is an unusually controllable lever — you can spend it on the hard tenth of your traffic and not the rest.
The third is the failure that the curve hides. Because the returns flatten rather than fall off a cliff, a long budget rarely looks harmful; it just stops paying. The real cost shows up as latency and spend on questions that never needed it. Setting a budget well is therefore a routing problem — deciding which questions deserve it — more than a tuning problem.
A picture of it
THE PICTURE #How to readThis is a schematic of a shape, not measured data — the numbers are illustrative and no real benchmark is being reported. Read only two features. The horizontal axis doubles at every step, so the early steep climb means each doubling of thinking is buying a large gain; the flattening on the right means later doublings buy progressively less, and the curve settles below the top of the chart because a budget cannot solve the problems whose difficulty is missing knowledge rather than missing search.
What became clearer
WHAT CLEARED #A model's weights hold capability; a thinking budget decides how much of that capability actually gets exercised on your question. Nothing is learned at inference — what is bought is search, paid for in tokens, with steeply diminishing returns and a hard ceiling set by what the model knows and by whether it can recognise its own mistakes. "More time" improves the answer for the same reason a longer clock improves a chess move, and stops helping for the same reason.
Where to go next
ONWARD #- How a system decides which questions warrant a long budget, and what it costs to get that routing wrong.
- Why sampling many short attempts and picking among them is sometimes a better use of the same compute than one long chain.
- What trains a model to use a budget well, and why an untrained model given the same tokens gains far less.
Key terms
TERMS #| Term | What it means |
|---|---|
| Test-time compute | computation spent when answering, as opposed to during training. |
| Thinking budget | a cap on the tokens a reasoning model may spend working before it replies. |
| Reasoning model | a model trained specifically to make long private working productive rather than merely longer. |
| Diminishing returns | the pattern where each doubling of a resource yields a smaller gain than the one before. |
Every term the collection defines is gathered in the glossary.