Penalized Regression (Ridge and Lasso)

Context: FIT2086_MOC · the stable alternative to all-or-nothing subset search in Model Selection and Information Criteria (AIC, BIC) · shrinks least-squares coefficients to trade a little bias for less variance (Bias-Variance Tradeoff (Underfitting vs Overfitting)) · $\lambda$ chosen by Cross-Validation · in R ➔ Penalized Regression in R (glmnet) · one-parameter derivation ➔ Shrinkage Estimator of the Mean

Quick Revision

• 🎯 Objective: minimise $\text{RSS}+\lambda {\sum }_{j}g({\beta }_j)$ on standardised predictors ➔ $\lambda$ sets model complexity ➔ pick $\lambda$ by CV, refit on all the data.
• 📦 Core Components: Ridge $g={\beta }_j^2$ ➔ stable, never exactly zero | Lasso $g=|{\beta }_j|$ ➔ sparse, performs variable selection.
• ⚠️ Key Constraint: ridge cannot select variables; lasso can, but biases large coefficients and struggles with correlated predictors.

📝 How It Works

1. Statistical Instability of Subset Selection

• Conventional methods ➔ all-subsets (score every combination), forward selection, backward selection — intuitive but expensive or unstable.
• Their problems ➔ statistically unstable (small data change ⟹ big model change) · multiple-testing false positives (Multiple Testing and the Bonferroni Correction) · all-subsets infeasible for moderate-to-large $p$ · stepwise hurt by correlated predictors and slow for large $p$.
• Diabetes experiment ➔ $n=354$, $p=10$ (AGE, SEX, BMI, BP, S1–S6); remove one random sample, search all $2^{10}=1024$ subsets for the smallest CV error, repeat 4 times (same CV partitioning rule each time).

Test	AGE	SEX	BMI	BP	S1	S2	S3	S4	S5	S6
1		✓	✓	✓	✓	✓	✓		✓
2		✓	✓	✓	✓	✓	✓
3	✓	✓	✓	✓	✓	✓	✓	✓
4	✓	✓	✓	✓	✓	✓	✓		✓

• Reading ➔ one observation of 354 flips AGE, S4, S5 in and out ➔ the all-or-nothing include/exclude decision makes the “best” model jump with tiny data changes.

2. Penalised Least Squares

• Objective ➔ penalise the coefficients instead of choosing subsets:

$$({\hat{\beta }}_0,{\hat{\mathbf{\beta }}}_{\lambda })=\arg \underset{{\beta }_0,\mathbf{\beta }}{\min }\left\{\text{RSS}({\beta }_0,\mathbf{\beta })+\lambda \sum _{j=1}^{p}g({\beta }_j)\right\}$$

• Contrast ➔ ordinary least squares minimises goodness-of-fit ($\text{RSS}$) only.
• Penalty choice ➔ $g(\cdot )$ increases with $|{\beta }_j|$; the sum runs over $j=\mathrm{1..}p$, so the intercept is not penalised — it is the baseline prediction, and shrinking it towards $0$ would bias every prediction without reducing complexity.
• Generalises ➔ logistic regression becomes penalised maximum likelihood (penalty added to the NLL).
• Standardise first ➔ penalising “large” coefficients only makes sense if size means strength, but ${\beta }_j$‘s size depends on the predictor’s scale ⟹ require ${\sum }_i{x}_{i,j}=0$ and ${\sum }_i{x}_{i,j}^2=n$ via $x_{i,j}^s=\frac{x_{i,j}-{\bar{x}}_j}{s_j}$ ⟹ larger $|{\beta }_j|$ means stronger association; ${\beta }_j=0$ means no association.

3. The Hyperparameter $\lambda$ and the Path

• $\lambda$ ➔ user-chosen hyperparameter controlling penalty strength.
• Limits ➔ $\lambda \to 0$ ⟹ $\hat{\mathbf{\beta }}\to$ least squares; $\lambda$ larger ⟹ $\hat{\mathbf{\beta }}$ smaller.
• Complexity dial ➔ varying $\lambda$ traces a path of models; moving towards small $\lambda$ the models become more complex.
• Choosing $\lambda$ ➔ (1) vary $\lambda$ over a grid; (2) for each, estimate prediction error by CV; (3) take the $\lambda$ with smallest CV error; (4) refit with that $\lambda$ on all the data. AIC/BIC can replace CV.
• Implementations ➔ R: the glmnet package; MATLAB: lasso(), lassoglm().

4. Ridge Regression

• Penalty ➔ ${\ell }_2$: $\lambda {\sum }_{j=1}^p{\beta }_j^2$ — a parabola in $\beta$, steeper for larger $\lambda$.
• Strengths ➔ very quick even for large $n$ and $p$ · very stable, low-variance estimates · well suited to multicollinearity.
• Weakness ➔ cannot set a coefficient exactly to zero ⟹ no variable selection.
• Path shape ➔ coefficients shrink smoothly towards zero as $\lambda$ grows.

5. Lasso Regression

• Penalty ➔ ${\ell }_1$: $\lambda {\sum }_{j=1}^p|{\beta }_j|$ — a V with its corner at $\beta =0$.
• Strengths ➔ efficient algorithms · coefficients can be exactly zero ⟹ a sparse estimator that performs variable selection.
• Weaknesses ➔ biased estimates for large coefficients · correlated predictors are problematic · can overfit (include unassociated predictors).
• Path shape ➔ coefficients hit zero one by one as $\lambda$ grows; reading the path right-to-left is the order predictors enter.

6. Why Penalisation Helps

• Advantages ➔ increased statistical stability · works when $n<p$ · better behaviour with correlated predictors (ridge particularly).
• Multicollinearity ➔ $\ge 2$ predictors highly correlated ⟹ hard to separate their relationships with the target ⟹ inflated variance in least squares, and dramatically more instability in stepwise methods.
• Bias–variance motivation ➔ if the linear model is correct, OLS is unbiased but can have high variance; shrinking towards zero deliberately adds bias to cut variance ⟹ a good $\lambda$ lowers ${\text{MSE}}_f={\text{bias}}^2+\text{variance}$.
• Irreducible error untouched ➔ ${\sigma }^2$ in prediction error does not depend on $\lambda$.
• Hand-derivable case (Studio 8) ➔ ridge on an intercept alone gives $\hat{\mu }(c)=\frac{n}{n+c}\bar{Y}$: bias $-\frac{c\mu }{n+c}$, variance $\frac{n{\sigma }^2}{(n+c{)}^2}$ ➔ Shrinkage Estimator of the Mean.
• Small-$n$ evidence (Studio 8) ➔ Pima, $n=100$, $p=59$: plain glm AUC $0.618$ · backward BIC $0.623$ · lasso $0.789$ · ridge $0.795$ ⟹ penalisation degrades gracefully where stepwise collapses; at $n=668$ lasso only matches stepwise ➔ Penalized Regression in R (glmnet).

⚖️ Core Decision Matrix

Method	Penalty	Exact zeros?	Stability	Correlated predictors	Main weakness
Subset / stepwise	none (discrete in/out)	yes	unstable	makes stepwise worse	infeasible or slow for large $p$
Ridge	$\lambda \sum {\beta }_j^2$	no	very stable	handles well	no variable selection
Lasso	$\lambda \sum |{\beta }_j|$	yes (sparse)	more stable than subsets	problematic	biases large coefficients; can overfit

When It Flips: need a reduced predictor set ➔ lasso; predictors heavily correlated and only prediction matters ➔ ridge.

📊 Exam Execution Trace & Applied Exercises

Manual Execution Trace

Lasso on the degree-20 polynomial example ($x,x^2,\dots ,x^{20}$, true order 5), decreasing $\lambda$:

$\lambda$	$\text{RSS}$	$\sum |{\hat{\beta }}_j|$	CV error	DF (non-zero terms)
$2.414$	$625.85$	$0.00$	$12.93$	$0$ (flat line)
$0.868$	$357.82$	$5.63$	$8.58$	$2$
$0.312$	$238.42$	$12.66$	$5.84$	$3$
$0.112$	$196.46$	$17.44$	$5.21$	$4$
$0.040$	$189.05$	$19.78$	$7.08$	$5$
$0.014$	$184.49$	$27.86$	$11.22$	$7$
$0.005$	$171.19$	$89.13$	$14.83$	$9$
$0.002$	$166.31$	$150.12$	$19.22$	$11$
$0.001$	$161.23$	$311.90$	$19.67$	$15$
$0.000$	$157.60$	$542.90$	$18.45$	$18$

• Read it ➔ $\text{RSS}$ falls monotonically as $\lambda \downarrow$ (in-sample fit always improves) while CV error is U-shaped ➔ selected $\lambda =0.112$ with 4 active terms; beyond it $\sum |{\hat{\beta }}_j|$ explodes and the curve wiggles (overfit).

⚠️ Common Mistakes

• 💡 Skipping standardisation ➔ the penalty then punishes predictors for their units, not their weakness.
• 💡 Choosing $\lambda$ by $\text{RSS}$ ➔ RSS always prefers $\lambda =0$; use CV (or AIC/BIC).
• 💡 Claiming ridge selects variables ➔ ridge shrinks but never zeroes; only lasso gives exact zeros.
• 💡 Reporting the CV-fold fit ➔ after picking $\lambda$, refit on all the data.

🧠 Active Recall

Why is all-subsets selection unstable, and why does shrinkage help?

Answer

• Short answer: Each predictor is either fully in or fully out, so a tiny data change can flip the choice and the whole model jumps; a penalty moves coefficients continuously, so small data changes give small coefficient changes.
• Why: Discrete vs continuous ➔ the diabetes experiment flipped AGE/S4/S5 after removing one of 354 points; ${\hat{\mathbf{\beta }}}_{\lambda }$ varies smoothly with the data and with $\lambda$.

Why can lasso set coefficients exactly to zero while ridge cannot?

Answer

• Short answer: The ${\ell }_1$ penalty keeps a constant pull towards zero even for tiny coefficients (its V has a corner at $0$), while the ${\ell }_2$ penalty’s pull vanishes as ${\beta }_j\to 0$ (flat-bottomed parabola).
• Why: Penalty shape ➔ $\lambda |{\beta }_j|$ vs $\lambda {\beta }_j^2$ — the lecture plots show the V and the parabola; the paths show lasso coefficients hitting zero and ridge ones only approaching it.

If OLS is unbiased, how can a biased penalised estimator predict better?

Answer

• Short answer: Error is ${\text{bias}}^2+\text{variance}$; shrinkage adds a little bias but can remove much more variance.
• Why: Bias–variance trade ➔ $\lambda$ controls the exchange; ${\sigma }^2$ is unaffected, so the whole gain comes from the variance term.