Time to first token
A Socratic walk-through of time to first token — reasoned out one step at a time, not lectured.
The question we started with
THE QUESTION #Why do two answers that take exactly the same time feel instant and broken?
Two systems answer the same question. Both finish in four seconds; a stopwatch cannot tell them apart. One begins putting words on the screen after a fifth of a second and lays them down steadily until it is done. The other shows a spinner for three and a half seconds and then paints the whole answer at once. Users call the first one fast and the second one broken.
If the total is identical, the difference is not in the machine's work — it is in when the user learns that the work is happening. So what is the user actually measuring?
Reasoning it through
REASONING #Begin with what the machine is doing, because the split is real and not merely a presentational trick.
A language model answers in two distinct phases. First it must read the prompt: every token of it is processed, and because they can all be attended to at once, this phase is a big dense matrix computation that saturates the arithmetic units. This is the prefill, and its cost grows with prompt length. Only when prefill completes can the first output token be produced. Then the model enters decode, where it produces one token at a time, each conditioned on all that came before. Each decode step needs comparatively little arithmetic but must stream the model's weights out of memory, so the limit here is memory bandwidth rather than compute.
Notice what that means. The gap before the first token and the pace of the tokens after it are governed by different physical bottlenecks. They are not two views of one number, and they can be tuned independently — which is precisely why one system can have a short first gap and another a long one at equal totals.
Now the human side. Time perception is not a stopwatch. Long-standing interface research — Doherty and Thadani's work at IBM in 1982, and the response-time thresholds Jakob Nielsen popularised — converges on the same shape: below about a tenth of a second an action feels instantaneous, up to about a second the user's train of thought survives, and past roughly ten seconds attention leaves entirely. But there is a second finding that matters more here: an interval filled with visible progress is judged shorter than an empty interval of the same length. An unmarked wait is not merely unpleasant; it is measured as longer by the person waiting.
There is a third ingredient specific to text. People read prose at something like 200 to 300 words per minute — roughly four or five words a second. If tokens arrive faster than that, the reader is never waiting on the machine; they are waiting on themselves. So beyond a certain streaming rate, making the model faster stops changing the experience at all.
Put the three together. What does the user experience as "the wait"? Only the first gap. After that, they are reading, and the remaining seconds are spent doing something rather than nothing.
The analogy
THE ANALOGY #Think of a kitchen with an open pass. You order, and within moments a waiter brings bread, then a glass of water, and you can see plates moving. Forty minutes later you leave, having felt looked after. In the second restaurant the same forty minutes pass in silence, at an empty table, with no acknowledgement until the whole meal arrives at once. The kitchens worked identically hard for identical durations, and only one of them will be described as slow.
bread genuinely is part of the meal you came for, whereas a stream of tokens is the same answer merely delivered in pieces — so unlike the restaurant, streaming adds nothing of substance, and its entire benefit is informational.
Clarifying the model
THE MODEL #The useful move is to stop treating latency as one number. Serving systems are normally judged on three:
Time to first token is queue time plus prefill. It is what the user experiences as the wait, and it is dominated by how long the prompt is and by how busy the server is. This is why a huge retrieved context is not free even when it improves the answer: every retrieved document lengthens prefill and therefore the perceived wait.
Inter-token latency is the pace once words are flowing. It only needs to comfortably exceed reading speed; beyond that, improvements are invisible to a reader, though they still matter enormously when the output is being consumed by a program rather than a person.
Total generation time is what actually matters when nobody is watching — a batch job, an agent step whose output feeds the next tool call. Here streaming buys nothing, and the honest optimisation target is throughput.
Two misconceptions to head off. First, streaming is not a way of making the system faster; it is a way of making it legible, and if the first token arrives late, streaming does not save you — a spinner and a two-second first token feel equally stalled. Second, low time to first token is not universally good: it can be bought by starting to speak before the system has retrieved or checked anything, which trades a genuine quality property for a perceptual one. A system that streams a confident opening and then contradicts itself has spent its credibility on the impression of speed.
There is also a real design tension worth naming honestly. Any step you insert before the first token — a safety check on the whole response, a reranking pass, a plan-then-write structure — moves you back toward the second restaurant. The usual resolution is to give the user something true to watch during that interval, such as the actual step being performed, rather than a generic spinner. That is not a trick: it is restoring the information the streaming would have carried.
A picture of it
THE PICTURE #How to readthe upper chain is what the server is doing and the lower pair is what the person sees. Trace the lower path: the user sits in the silent state through both queueing and prefill, and leaves it the instant the first token lands — everything after that is spent reading, not waiting. The self-loop on the decoding state is the per-token pace, which only needs to outrun the reader, whereas shortening the silent state is the only change the user experiences as speed.
What became clearer
WHAT CLEARED #Total latency is a property of the system; perceived latency is a property of the first gap. Because prefill and decode are limited by different resources, those two numbers move independently — so the same four seconds can be spent almost entirely before the user learns anything, or almost entirely with them reading. Optimising the wrong one produces systems that benchmark well and feel terrible.
Where to go next
ONWARD #- Why prefill is compute-bound and decode is memory-bandwidth-bound, and what that implies for hardware choice.
- Prompt caching: reusing the prefill work for a shared prefix, and what it does to first-token time.
- Whether streaming still helps when the consumer is an agent loop rather than a reader, and where partial output is actively dangerous.
Key terms
TERMS #| Term | What it means |
|---|---|
| Prefill | processing the entire input prompt in one pass to produce the state from which generation begins; the main controllable component of the first-token wait. |
| Decode | generating output one token at a time, each step depending on the last; limited chiefly by memory bandwidth. |
| Time to first token | elapsed time from request to the first visible output; the interval users perceive as the wait. |
| Inter-token latency | the interval between successive output tokens, i.e. the streaming pace. |
| Doherty threshold | the observation from IBM's 1982 work that response times under roughly 400 milliseconds keep a user engaged rather than merely tolerant. |
Every term the collection defines is gathered in the glossary.