Regularization

Fitting a model by minimizing training error alone gives it every incentive to contort itself toward the training sample. Regularization changes the objective: instead of minimizing error alone, the model minimizes error plus a penalty that grows with the size of its parameters. Large, elaborate fits must now justify themselves against that cost, so the model settles for a simpler one unless the data really supports more.

The two classical penalties behave differently. The L2 penalty, the sum of squared coefficients, gives ridge regression. Its gradient is proportional to the coefficient itself, so it shrinks every coefficient smoothly toward zero without reaching it; correlated predictors tend to share the influence between them. The L1 penalty, the sum of absolute values, gives the lasso. Its gradient has constant magnitude, which is enough to drive coefficients exactly to zero, so the fitted model performs variable selection and yields a sparse, more interpretable result.

Both are expressions of the bias-variance trade-off. Shrinking coefficients away from their least-squares values introduces bias, since the fitted model is no longer the best possible fit to the training data. In exchange, the fit becomes far less sensitive to which observations happened to be sampled. When predictors are numerous or collinear, that trade is usually strongly favourable.

The idea generalizes well beyond linear models. In neural networks the same L2 penalty appears as weight decay, and other devices, dropout, early stopping, data augmentation, act as regularizers even though they add no explicit penalty term. All of them limit how precisely the model can conform to its training sample.

Consider a regression with many predictors, several of them strongly correlated. Ordinary least squares can produce large coefficients of opposing sign that cancel one another out: the fit is good on the training data but wildly unstable, since a slightly different sample would produce very different coefficients.

Applying an L2 penalty suppresses that behaviour. Large opposing coefficients are expensive, so the fit distributes the influence more evenly across the correlated predictors and the coefficients become far more stable across samples.

Applying an L1 penalty of comparable strength does something qualitatively different: it drives most of the correlated predictors’ coefficients to exactly zero and keeps one. The result is sparser and easier to interpret, but which predictor survives can be somewhat arbitrary when the correlated group is nearly interchangeable.