Regularization

Shrinkage

2026-10-06

Learning Outcomes

  • Regularization

  • Ridge Regression

  • LASSO Regression

Why Regularization?

  • The additive linear models (Linear, Logistic, GLM) can be limiting by assuming a line.
    • However, these models tend to perform better, and more efficiently than other models.
    • They also allow for inferential procedures.
  • Regularization will improve prediction accuracy and model interpretability.

Caution

Statisticians warn against using regularization techniques as it threatens inference. If one must be done, Ridge Regression is preferred.

Regularization

Shrinkage methods are techniques that will reduce a full parameterized model (a high number of predictors) to a lower parameterized model (a smaller number of predictors).

Ridge Regression

Ridge regression incorporates a shrinkage penalty term to the least squares formula. The shrinkage penalty term will reduce the \(\beta\) coefficients towards 0 based on a penalty parameter (\(\lambda\))

Ridge Regression

\[\sum^n_{i=1}\left(Y_i-\beta_0 +\sum^p_{j=1}X_{ij}\beta_j\right)^2 + \lambda\sum^p_{j=1}\beta_j^2\]

Ridge Regression

\[ \ell(\boldsymbol \beta|\boldsymbol Y;\boldsymbol X) = \sum^n_{i=1}\log\left(f\left\{Y_i;\boldsymbol \beta, X_i\right\}\right) + \lambda\sum^p_{j=1}\beta_j^2 \]

Why does Ridge Regression Work?

The inclusion of a tuning parameter \(\lambda\) helps control the bias-variance tradeoff. An increase in \(\lambda\) will begin to reduce the variance, while slightly increasing the bias in the prediction of new data.

LASSO

Least Absolute Shrinkage and Selection Operator (LASSO) is known as a shrinkage method which forces \(\beta\) coefficients that do not have a significant predict power towards and possibly equal to 0.

LASSO

\[\sum^n_{i=1}\left(Y_i-\beta_0 +\sum^p_{j=1}X_{ij}\beta_j\right)^2 + \lambda\sum^p_{j=1}|\beta_j|\]

LASSO

\[ \ell(\boldsymbol \beta|\boldsymbol Y;\boldsymbol X) = \sum^n_{i=1}\log\left(f\left\{Y_i;\boldsymbol \beta, X_i\right\}\right) + + \lambda\sum^p_{j=1}|\beta_j| \]

Why Ridge or LASSO?

Each method is capable on identifying the optimum MSE for the Bias-Variance trade-off scenario. This will lead to a lower prediction error. The key is to find the optimal penalty parameter. This can be done with a Cross-Validation technique (next lecture).

Ridge Regression in Python

Ridge Regression Notebook

LASSO in Python

Lasso Regression Notebook

Resources