Regularization: Ridge and Lasso

Least squares has one objective: make RSS as small as possible on the training data. The Bias-Variance Tradeoff showed why that is not the same as being right, and Logistic Regression showed coefficients running away to $\pm\infty$ when nothing restrains them.

Regularization adds a second term to the objective that makes large coefficients expensive. The result is deliberately worse on the training data and often much better on new data.

A. Ridge regression

Ridge minimises RSS plus a penalty proportional to the sum of squared coefficients:

$$\sum_{i=1}^{n}\Big(y_i - \beta_0 - \sum_{j=1}^{p}\beta_j x_{ij}\Big)^2 + \lambda\sum_{j=1}^{p}\beta_j^2 \;=\; \mathrm{RSS} + \lambda\sum_{j=1}^{p}\beta_j^2 .$$

The tuning parameter $\lambda \ge 0$ controls the trade. At $\lambda = 0$ the penalty vanishes and we recover least squares. As $\lambda \to \infty$ the penalty dominates and every coefficient is driven towards zero.

Two details matter. The intercept $\beta_0$ is not penalised - it only sets the overall level and shrinking it would be meaningless. And because the penalty acts on coefficient magnitudes, predictors must be standardised first: otherwise measuring a variable in grams rather than kilograms would change how hard it is penalised, which is clearly wrong.

B. Lasso

The lasso replaces the squared penalty with an absolute-value penalty:

$$\mathrm{RSS} + \lambda\sum_{j=1}^{p}\big|\beta_j\big| .$$

Ridge uses an $\ell_2$ penalty, the lasso an $\ell_1$ penalty. That single change has a consequence out of proportion to its size: the $\ell_1$ penalty forces some coefficients to be exactly zero once $\lambda$ is large enough. Ridge shrinks coefficients towards zero but, except in the limit, never to zero.

A model with exact zeros ignores those predictors entirely, so the lasso performs variable selection and yields sparse models - generally far easier to interpret than a ridge fit that keeps all $p$ predictors with small coefficients.

C. Watching them shrink

Using the five-observation dataset from Linear Regression from First Principles ($x = 1..5$, $y = 2,4,5,4,5$), whose least-squares fit was $\hat y = 2.2 + 0.6x$:

Both methods pull the slope down as $\lambda$ grows, and the intercept rises to compensate - it is unpenalised, so it absorbs the level.

The difference in how they approach zero is the whole point. Ridge divides: its slope is $6/(10 + \lambda)$, so at $\lambda = 20$ it is $0.2$, a third of the original, but alive. The lasso subtracts: its slope is $(6 - \lambda/2)/10$, which hits exactly $0.0$ at $\lambda = 12$ and stays there. From then on the model is just $\hat y = 4.0 = \bar y$ - the predictor has been switched off completely.

These numbers fit the raw $x$, not a standardised one. With a single predictor that only changes which $\lambda$ produces each row, not the shape of either path.

Running it prints every row of the table, including lambda=12 ridge 0.2727 3.1818 lasso 0.0000 4.0000. Mind the rescaling: scikit-learn's Lasso divides RSS by $2n$, so its alpha is $\lambda/(2n)$. Passing $\lambda$ straight in would fit each lasso row at a penalty ten times larger than the ridge row beside it.

A caveat on this demonstration. With one predictor and five observations there is nothing to select - sparsity is only interesting when some predictors are genuinely irrelevant and the lasso can switch those off while keeping the rest. The table shows the mechanism honestly, not the setting where it pays.

D. Why the penalties behave differently

The geometric account is the clearest. Both problems can be written as: minimise RSS subject to a budget on coefficient size -

$$\sum_j \beta_j^2 \le s \quad\text{(ridge)}, \qquad \sum_j |\beta_j| \le s \quad\text{(lasso)} .$$

In two dimensions the ridge budget is a circle and the lasso budget a diamond with corners on the axes. The solution is where the elliptical contours of RSS first touch the budget region.

A circle has no corners, so the touch point almost never lands exactly on an axis - both coefficients stay non-zero. The diamond's corners sit on the axes, and expanding contours hit a corner readily. A corner means one coefficient is exactly zero. In higher dimensions the diamond has ever more corners, edges, and faces, and hitting one zeroes out several coefficients at once.

E. Why this reduces error

Regularization is the bias-variance tradeoff applied deliberately. Shrinking coefficients away from their least-squares values introduces bias: the estimator is no longer centred on the truth. But it also makes the fit far less sensitive to which particular sample you drew, which reduces variance.

When variance falls faster than squared bias rises, total expected error falls. That is most likely when least squares is unstable: many predictors relative to observations, or strongly correlated predictors. In the extreme case $p > n$, least squares has no unique solution at all. Ridge still has exactly one; the lasso still has solutions, though they need not be unique (two identical columns can split their weight in many ways at the same cost).

The bias grows monotonically with $\lambda$ and the variance falls monotonically, so there is an interior optimum - and it cannot be found from training error, which always prefers $\lambda = 0$. It is chosen by cross-validation over a grid of $\lambda$ values.

F. Choosing between them

Neither dominates. It is an empirical question about the problem, settled by cross-validating both. The last row is a practical gotcha: with a group of correlated predictors the lasso's arbitrary choice can be unstable across resamples, which is why elastic net - combining both penalties - exists.

Key takeaways

• Regularization adds a penalty on coefficient size to RSS, controlled by $\lambda$, trading training fit for stability.
• Ridge ($\ell_2$) shrinks coefficients towards zero; lasso ($\ell_1$) sets some exactly to zero, performing variable selection.
• Predictors must be standardised, and the intercept is never penalised.
• The difference is geometric: a circular budget has no corners, a diamond has corners on the axes.
• It works by trading a little bias for a large variance reduction; $\lambda$ must be chosen by cross-validation, never by training error.

What's next

Every model so far has been a single global formula. A different strategy is to carve the predictor space into regions and predict a constant in each - which handles interactions and non-linearity without anyone specifying them in advance. That is Decision Trees and Ensembles.