Ridge, lasso and elastic net

Least squares is unbiased, and with many correlated inputs that is a weakness: its variance is huge. Deliberately shrinking coefficients towards zero adds a little bias and removes a lot of variance. With hundreds of noisy signals, that is usually a winning trade.

Why shrink at all?

Test error. Total error is bias² + variance (lesson 9). Plain regression sits where a bit more bias buys a big cut in variance.
Interpretability. Knowing which 5 of 200 inputs matter is worth a lot.

Subset selection

The bluntest tool keeps a subset of $k$ inputs and drops the rest.

Forward stepwise: start empty; repeatedly add the input that improves the fit most.
Backward stepwise: start full; repeatedly remove the input that contributes least.
Forward stagewise: start at zero; repeatedly find the input most correlated with the current residual and nudge its coefficient. It takes many more steps but behaves well in very high dimensions.

For up to about 30 inputs, the leaps-and-bounds algorithm finds the best subset of each size efficiently.

Ridge regression (L2)

Penalise the squared size of the coefficients:

\hat\beta^{\text{ridge}} = \arg\min_\beta \Big\{\sum_{i=1}^N \Big(y_i - \beta_0 - \sum_{j=1}^p x_{ij}\beta_j\Big)^2 + \lambda\sum_{j=1}^p \beta_j^2\Big\} = (X^\top X + \lambda I)^{-1}X^\top y .

$X^\top X + \lambda I$ is always invertible, which was ridge’s original motivation. As $\lambda$ grows the effective degrees of freedom fall smoothly from $p$ towards 0. Ridge never sets a coefficient exactly to zero; it shrinks the least important directions (the last principal components, lesson 12) the most.

Lasso (L1)

Penalise absolute size instead:

\hat\beta^{\text{lasso}} = \arg\min_\beta \Big\{\sum_{i=1}^N \Big(y_i - \beta_0 - \sum_{j=1}^p x_{ij}\beta_j\Big)^2 + \lambda\sum_{j=1}^p |\beta_j|\Big\}.

No closed form: it is a quadratic programme, solved in practice by coordinate descent or least-angle regression. The payoff is sparsity: as $\lambda$ grows, coefficients hit exactly zero and stay there, so lasso does selection and shrinkage in one step.

Figure 11.1The RSS bowl and the penalty prism

Penalised regression is the lowest point of the RSS bowl that stays inside the penalty region. For lasso the region is a diamond prism, and the bowl usually first touches it on an edge sitting on an axis: a coefficient of exactly zero. Switch to ridge and the round cylinder only shrinks.

Figure 11.2The same picture from above

Penalised regression is the same as minimising RSS inside a budget region. The orange ellipses are RSS contours around the least-squares answer; the solution is where the smallest one first touches the blue region. The L1 diamond has corners on the axes, so the touch point is often a corner: a coefficient of exactly zero. Drag the orange dot and change the penalty.

Figure 11.3Coefficient paths

Six standardised inputs, three of which matter. From left (heavy penalty) to right (none), lasso switches coefficients on one at a time and keeps the useless ones at zero; ridge shrinks all of them smoothly. Elastic net is in between.

Three ways to shrink

With orthonormal inputs each method is a simple function of the least-squares estimate $\hat\beta_j$ :

Method	Estimate	Behaviour
Best subset of size $M$	$\hat\beta_j \cdot \mathbf 1\{\lvert\hat\beta_j\rvert \ge \lvert\hat\beta_{(M)}\rvert\}$	hard threshold: keep or drop
Ridge	$\hat\beta_j / (1+\lambda)$	proportional shrinkage
Lasso	$\operatorname{sign}(\hat\beta_j)\,(\lvert\hat\beta_j\rvert - \lambda)_+$	soft threshold: subtract, stop at 0

Figure 11.4Hard, proportional and soft thresholds

The dashed diagonal is “leave the estimate alone”. Raise λ to see each rule bite.

Elastic net and friends

Ridge and lasso are the $q = 2$ and $q = 1$ cases of an $L_q$ penalty $\lambda\sum|\beta_j|^q$ ( $q = 0$ is subset selection). The best $q$ in practice is often between 1 and 2, which elastic net approximates with a blend:

\lambda \sum_{j=1}^{p}\Big(\alpha\beta_j^2 + (1-\alpha)|\beta_j|\Big).

Least-angle regression (LAR) grows coefficients continuously, always moving the inputs most correlated with the residual. With one modification (drop a variable when its coefficient crosses zero) it traces the entire lasso path for the cost of one least-squares fit, and after $k$ steps the fit has exactly $k$ degrees of freedom.

In code

import numpy as np
from sklearn.linear_model import Ridge, Lasso, ElasticNet
from sklearn.preprocessing import StandardScaler
rng = np.random.default_rng(9)

N = 80
z = rng.standard_normal((N, 6))
X = np.c_[z[:, 0], 0.75 * z[:, 0] + 0.66 * z[:, 1], z[:, 2], 0.8 * z[:, 2] + 0.6 * z[:, 3], z[:, 4], z[:, 5]]
y = X @ np.array([3, 0, 1.5, 0, 0, -2]) + 1.6 * rng.standard_normal(N)
X = StandardScaler().fit_transform(X)

for lam in (0.05, 0.3, 1.0):
    print(lam,
          np.round(Lasso(alpha=lam).fit(X, y).coef_, 2),         # zeros appear
          np.round(Ridge(alpha=lam * N).fit(X, y).coef_, 2))      # all shrink, none zero

Check yourself

You have 1,000 candidate features and believe only about 20 of them matter. Which penalty fits that belief best?