Double descent

Why does adding still more parameters to an already overfitted model make its predictions better again?

The classical account of overfitting is not wrong, and this collection sets it out elsewhere: add flexibility, training error falls, test error falls and then rises, so find the sweet spot and stop there.

Yet the models that work best now sit far past that peak. They have more parameters than training examples, they fit their training data exactly, and they generalise well anyway. So either the classical curve is false, or it was measured on an axis that stopped too early. What would tell you the difference?

Start by extending the plot rather than arguing. Keep the data fixed, keep growing the model, record test error the whole way. What appears is a curve that falls, rises to a peak, and falls again — often below the first minimum. Belkin and colleagues named the shape double descent in 2019; Nakkiran and colleagues documented it in modern deep networks.

The peak does not sit anywhere arbitrary. It sits at the interpolation threshold — roughly where the model has just barely enough capacity to fit the training set exactly. Why should the worst model be the one that has just enough?

Take the case where the algebra is honest and small: least squares with n examples and p features. When p is well under n the system is over-determined; the fit is a compromise, and it is stable. When p exceeds n there are infinitely many coefficient vectors that reproduce the data exactly. But at p equal to n there is generically exactly one — a square system with a unique solution and no freedom at all in which solution you take.

That uniqueness is the trap. A square matrix drawn from data is typically ill-conditioned: its smallest singular value sits near zero, and inverting it multiplies the coefficients by one over that number. So the single fit that exists is achieved with enormous coefficients, and enormous coefficients mean a function that swings violently between the points it passes through. That is the variance term of the classical picture, at its maximum, and it is a fact of linear algebra rather than a story about learning.

Now walk past the threshold. With p greater than n, exact fits are plentiful, and the question changes from can it fit to which fit does it choose. Gradient descent from a small initialisation does not pick arbitrarily: it converges to the interpolating solution of smallest norm. And the more parameters you add, the larger the set of exact fits, so the smallest-norm member of that set gets smaller still. The function keeps passing through every training point while becoming smoother. Error descends a second time.

So the variable was never capacity on its own. It was capacity relative to the constraint the data imposes, together with the rule that selects among the fits — and classical statistics worked in the region where nothing selects because nothing is free.

How would we know this account is wrong? It makes sharp claims. The peak is pinned to the interpolation threshold, not to any model size — so make the data harder to fit by adding label noise, and the threshold should move and the peak sharpen. It does. The second descent comes from the optimiser making a min-norm choice for you — so supply that preference explicitly, with well-tuned ridge regularisation, and the peak should flatten or vanish, because you never depend on the accident of a unique fit. It does that too. The refuting observation would be a peak that stays put when you change the number of training examples at fixed model size, or one that survives optimally tuned regularisation while sitting at a capacity unrelated to the training-set size. Either would say the mechanism is something other than ill-conditioning at the threshold.

Three refinements hold the steps together.

First, this does not repeal the bias-variance account. That account is correct on its own axis; double descent says the axis was truncated. Both curves are the same curve.

Second, interpolating the training data is not identical to fitting the noise, once there is a choice about how to interpolate. A model with slack can pass through a noisy point while remaining smooth almost everywhere; a model with no slack cannot. The honest caveat is that why deep networks land on the benign choice is an open research question — the minimum-norm implicit-bias story is the leading account and is rigorous for linear and random-feature models, but its extension to deep networks is an argument in progress, not a settled result.

Third, none of this is a licence to stop validating. The peak's location depends on your data, your noise level and your architecture, so "just make it bigger" can land you exactly on the peak. And the second descent is typically slow: past the threshold you buy improvement with large multiples of compute, which makes it an economic question about the cost of parameters and data rather than a free lunch.

• Grokking, where the same generalisation arrives late along the training-time axis rather than the capacity axis.
• Benign overfitting: the conditions under which an exact fit to noisy data provably costs nothing.

Every term the collection defines is gathered in the glossary.