Foundation model economics

Why did building one model per task give way to adapting a single expensive one?

For most of machine learning's practical history the answer to "we need a spam classifier" was: gather spam, train a spam model. Need sentiment? Train a sentiment model. Each task got its own dataset, its own architecture, its own training run, and each was modest enough that a team could own one.

That arrangement has largely collapsed. Now the answer is to take one enormous model that nobody in the room trained, and adapt it. What made the small, targeted, well-understood approach lose to the expensive general one? It cannot simply be that the big model is better, because a purpose-built classifier on a narrow task often still is.

Start with a cost question rather than a quality one. Suppose you have fifty tasks. Under the old approach, what does the fiftieth cost? Roughly what the first cost — its own data collection, its own labelling, its own training, its own evaluation. Costs scale with the number of tasks, and each task's cost is dominated by labelled data for that task, which is the expensive part, because a person has to produce it.

Now think about what a general model is trained on. Not labels — raw text, which already exists in quantity and costs nothing to have been written. The training objective, predicting what comes next, generates its own supervision from the text itself. So the two approaches are not just different in scale; they draw on completely different supplies. One is limited by how much annotation you can buy, the other by how much text there is and how much compute you can point at it.

Here is the pivot. If pretraining on unlabelled text produces representations that transfer — and this is the empirical bet that turned out to be right — then the labelled data for a specific task no longer has to teach the model language. It only has to teach it this task, on top of a model that already handles the language part. So the same annotation budget goes vastly further, and tasks that could never justify a model of their own suddenly can.

Then follow the cost structure to its conclusion, because the economics does the rest. Pretraining is a large fixed cost, paid once, independent of how many tasks or customers use the result. Adaptation is a small marginal cost, paid per task. What happens to an industry whose costs look like that? The same thing that happens with a bridge or a chip fabrication plant: the fixed cost is only bearable if it is spread over enormous use, which favours a few large builders and many small adapters, and makes it irrational for the fiftieth team to pay the fixed cost again for their own bridge.

Watch how adaptation itself got cheaper, because that is the part that made the shift complete. In the 2018 pretrain-and-fine-tune era, adapting meant training and storing a full copy of the model per task — cheap next to pretraining, but not free. Then parameter-efficient methods such as LoRA, introduced in 2021, let a task be captured in a small set of extra weights beside a shared frozen model. And then, with GPT-3 in 2020 and what followed, adaptation for many tasks became writing a prompt: no training run at all, no artefact to store, a marginal cost measured in minutes. The Stanford group that coined "foundation model" in August 2021 named exactly this property — one model, adapted many ways, with the adaptation no longer the hard part.

One honesty note: none of this says the general model is more accurate on any given narrow task. On a well-defined problem with plenty of labels, a small specialised model can match or beat it, and run far cheaper at inference. The general model wins on the portfolio, not on the individual bet — and where inference cost dominates, distilling a small specialist back out of the big model is now the common answer, which is the old approach returning as a downstream step rather than a starting point.

Three refinements.

First, the causation is easy to state backwards. It was not that models got big and therefore general. It was that a training objective which needs no labels could absorb far more data than any labelled task could, and the generality came out of that. Scale was the means; the absence of an annotation bottleneck was the reason scale was possible at all.

Second, the shift moved who pays for what, not merely how much. Under the old model each team paid a small fixed cost and owned the result. Under the new one, the fixed cost sits with a handful of organisations and everyone else pays marginal costs per use, forever. That is a different dependency structure, and it explains the strategic anxiety around foundation models better than any argument about capability does.

Third, the misconception that follows most naturally from all this: that adaptation is cheap therefore easy. The training cost fell; the evaluation cost did not. A team that no longer trains anything still has to work out whether the adapted behaviour is right on their data, and that work is roughly unchanged from the old world. The expensive part quietly migrated from fitting the model to knowing whether it works.

• Why distilling a small specialist out of a large general model has revived task-specific models at the inference end.
• How evaluation became the dominant remaining cost, and what that does to team structure.
• What the same fixed-cost logic predicts about open-weight models changing who bears the fixed cost.

Every term the collection defines is gathered in the glossary.