Overfitting

Why can a model that predicts its training data perfectly be useless on anything new?

A model scores 100% on the data it was built from. That sounds like the best possible outcome, and the instinct is to ship it. But consider what the score is actually measuring: how well a curve reproduces the very points used to place it. Is there any reason that number should predict performance on a point the curve has never seen? Put that way, the perfect score starts to look less like evidence and more like a tautology.

Take the cleanest case. You have ten points, each with an x and a y, and you want a curve through them. A straight line will miss most of them. A polynomial of degree nine, however, passes exactly through all ten — this is not a coincidence but a fact of algebra: ten coefficients, ten equations, one solution. Training error: zero.

Now look between the points. The degree-nine curve swings violently, shooting far above and below the data in the gaps, and its behaviour just outside the range is wild. Ask the honest question: which curve would you rather use to guess y at a new x? Almost certainly the line, which fits nothing exactly.

So what went wrong? Assume, as is nearly always true, that each observed y is some underlying pattern plus noise — measurement error, sampling variation, the influence of things you did not record. A model with enough freedom cannot tell those two parts apart. It fits the pattern and the noise, because both are equally present in the data. The noise, however, is different next time, so everything the model learned about it is not merely useless but actively wrong.

That gives us the decomposition. Expected error on new data splits into three pieces: bias, the error from a model too rigid to represent the real pattern; variance, the error from a model so flexible that it changes substantially when you resample the data; and irreducible noise, which no model removes. The straight line is high bias, low variance — fit it to a different ten points and you get roughly the same line. The degree-nine polynomial is low bias, high variance — a different ten points produce a wildly different curve. Overfitting is the variance term dominating.

Now the second question, which is quieter but more important: why is training error not simply a bad estimate, but a systematically optimistic one? Because the parameters were chosen to minimise it. The same data both selected the model and graded it. Any accident in that data that the model could exploit, it did. So training error is biased downward, and the bias grows with the model's flexibility — which is exactly backwards from what you want a diagnostic to do.

The fix follows directly: grade on data the fitting never touched. Split into training, validation, and test. Train on the first. Use the second to compare candidates — how many degrees, how much regularization, when to stop. Note that this makes validation error slightly optimistic too, because you selected on it; hence the third set, touched once, at the end. Cross-validation rotates the validation role through the data when you cannot afford to hold much out.

And how do you actually restrain a model? Not usually by shrinking it, but by penalising complexity within it. Regularization adds a cost for large coefficients: ridge shrinks them toward zero, lasso pushes many exactly to zero. Early stopping halts training when validation error turns upward, before the fit begins consuming noise. More data helps directly — noise averages out, pattern does not. Each of these buys a little bias to remove a lot of variance, and the right amount of that trade is itself decided on held-out data.

Several refinements connect these steps.

Overfitting is not caused by "too many parameters" as such, but by effective flexibility relative to the amount and cleanliness of the data. The same model can overfit on 100 rows and underfit on 10 million. And the opposite failure is real: a model too constrained misses the pattern entirely, and its error is just as large — it merely looks more respectable, because it fails on training data too and is therefore honest about it.

A subtler trap is repeated use of the same held-out set. Evaluate 200 variants against one test set and pick the winner, and you have used it for selection; its score is now optimistic in exactly the way training error is. This is why leaderboard scores drift away from true performance, and why a test set is spent once.

The most important honesty caveat: the classical picture — error falling, then rising, as complexity grows — is not the whole story. Very large neural networks with far more parameters than training examples often generalize well despite interpolating their training data, and error can fall again past the point where the classical curve says it should rise. This "double descent" behaviour is documented, and why it happens is an active research question rather than a settled one. The bias-variance framing remains the right first model, and it is not the last word.

• Cross-validation schemes, and when a single held-out split is not enough.
• Double descent, and why heavily over-parameterised models interpolate without collapsing.
• Data leakage: how information from the test set sneaks into training through preprocessing.

Every term the collection defines is gathered in the glossary.