Orchestration

Why is the model often the least important part of a working AI product?

Two teams build the same thing on the same model. One product is useful and one is a demo that impressed people once. The model is identical, so whatever separated them was not the model.

That observation is easy to nod along to and hard to take seriously, because the model is the part that seems miraculous. So let us make the claim uncomfortable and precise: in most working systems, the model is the component you could swap for a competitor's over a weekend, and everything you could not swap is where the product lives. Why would that be?

Begin with a simple test of importance: what happens if you remove or replace a part? Replace the model with a comparable one from another vendor and, in a well-built system, you re-run your evaluations, adjust some prompts, and continue. Remove the retrieval layer that finds the right three documents and the same model answers confidently about the wrong contract. Which of those is the system more sensitive to?

That asymmetry has a reason, and it is worth naming carefully. The model is general — it is built to be adequate at everything and specific to nobody. But your users have one situation, one set of documents, one definition of a correct answer. The distance between "a generally capable model" and "answers this company's questions correctly" is exactly the work the model cannot do, because it has never seen the material and never will.

So what closes that distance? Ask what the model needs at the moment it generates. It needs the right context in front of it, which means something must decide what to retrieve and what to leave out. It needs the task framed in a way that matches how it will be judged. And the output needs to be checked, or routed, or refused. Notice that all three are decisions about what surrounds the call — and none of them is made by the model.

Then there is the part that is invisible in a demo and dominant in production. What does the system do when the model returns something malformed, or the provider returns an error, or the answer is fluent and wrong? A demo has no answer to those questions because they did not come up in the four minutes it ran. A product must answer all three, every time, and the answers are retries, fallbacks, validation, logging, and a way to find out tomorrow that quality slipped today.

And the hardest of them: how do you know the system is working? That is evaluation, and it does not come with the model. Someone has to decide what a good answer is for this task, assemble cases, and measure. Teams routinely find this is the single largest piece of work in the project — larger than the integration, larger than the prompting — and it is entirely their own, because "good" is defined by their users and not by any benchmark.

The claim needs a boundary, though, or it becomes false. The model sets a ceiling. No orchestration turns a weak model into one that can do a task beyond it, and when a genuinely more capable model arrives, some carefully engineered scaffolding becomes unnecessary overnight — a real and repeated pattern, and a reason not to over-invest in workarounds for a specific model's weaknesses. The honest form of the claim is not that the model does not matter. It is that the model is necessary and not differentiating: everyone can buy the same one, so it cannot be what distinguishes your system, while your data, your context assembly, and your definition of correct are not for sale.

Three refinements.

First, the misconception being corrected is about where difficulty lives, not where value lives. Nobody thinks the model is easy to build — it is the hardest engineering in the stack. The point is that its difficulty was already paid for by someone else and is available to your competitor at the same price. Difficulty and differentiation are different axes, and confusing them is what produces a project plan with the prompt at the centre and the evaluation set in a footnote.

Second, orchestration is not a synonym for a framework. It names the set of decisions — what goes into the context, in what order steps run, what happens on failure, what gets measured, what a person must approve — and those decisions exist whether you adopt a library or write the code yourself. A framework changes who writes the plumbing, not who decides.

Third, and this is the practical consequence: the parts of the system that are yours are also the parts you can improve. You cannot make the model better. You can make retrieval find the right document, tighten the schema so bad output is caught, and grow the evaluation set until it tells you the truth. The distribution of effort should follow the distribution of what is actually under your control.

• How to build an evaluation set that reflects real usage rather than the cases that were easy to write down.
• Which scaffolding is worth building given that a stronger model may make it redundant.
• Where retrieval quality, rather than model quality, sets the ceiling on an answer's accuracy.

Every term the collection defines is gathered in the glossary.