Least Squares as Maximum Likelihood

Context: FIT2086_MOC · the bridge that licenses everything downstream — least squares is maximum likelihood under normal errors, which is why a likelihood-based criterion can score regression models (Model Selection and Information Criteria (AIC, BIC)) and why residuals get a normal predictive distribution

Quick Revision

• 🎯 Objective: put ${\epsilon }_i\sim N(0,{\sigma }^2)$ on the regression errors ➔ the negative log-likelihood becomes $\frac{n}{2}\log (2\pi {\sigma }^2)+\frac{\text{RSS}}{2{\sigma }^2}$ ➔ minimising NLL in $\mathbf{\beta }$ = minimising RSS.
• ⚡ Key Constraint: ${\sigma }^2$ only scales the RSS term, so it drops out of the $\mathbf{\beta }$ argmin — the equivalence holds for any ${\sigma }^2$, but only under the normal error assumption.

📝 Core

• Error form of the model ➔ $Y_i={\beta }_0+{\sum }_{j=1}^p{\beta }_j{x}_{i,j}+{\epsilon }_i$ with ${\epsilon }_i\sim N(0,{\sigma }^2)$, independent across $i$.
• Equivalent conditional statement ➔ $Y_i\mid x_{i,1},\dots ,x_{i,p}\sim N({\beta }_0+{\sum }_j{\beta }_j{x}_{i,j},{\sigma }^2)$ — the mean moves with the predictors, the variance does not.
• Residuals estimate the errors ➔ $e_i=y_i-{\hat{y}}_i$ is the observable stand-in for the unobservable ${\epsilon }_i$.
• What the equivalence buys ➔ a fitting rule (least squares) is upgraded to a probability model, unlocking $p$-values on ${\hat{\beta }}_j$, predictive distributions, and penalised-likelihood model selection.
• ML variance estimate ➔ ${\hat{\sigma }}_{ML}^2=\frac{\text{RSS}({\hat{\beta }}_0,\hat{\mathbf{\beta }})}{n}$ ➔ underestimates the true variance (same bias story as ${\hat{\sigma }}_{ML}^2$ in Estimator Quality (Bias, Variance, MSE)).
• Unbiased variance estimate ➔ ${\hat{\sigma }}_u^2=\frac{\text{RSS}({\hat{\beta }}_0,\hat{\mathbf{\beta }})}{n-p-1}$ ➔ divisor is $n$ minus the $p+1$ fitted coefficients, not $n-1$.

🧮 Proof Blueprint

Theorem. If ${\epsilon }_i\stackrel{\text{iid}}{\sim }N(0,{\sigma }^2)$ then $\arg {\min }_{{\beta }_0,\mathbf{\beta }}L(\mathbf{y}\mid {\beta }_0,\mathbf{\beta },{\sigma }^2)=\arg {\min }_{{\beta }_0,\mathbf{\beta }}\text{RSS}({\beta }_0,\mathbf{\beta })$ — the ML and LS estimates coincide.

Strategy. Write the iid normal likelihood, collapse the product of exponentials, recognise RSS in the exponent, take $-\log$, and note ${\sigma }^2$ is a positive constant factor on the only $\mathbf{\beta }$-dependent term.

Derivation.

$$\begin{array}{rl}p(\mathbf{y}\mid {\beta }_0,\mathbf{\beta },{\sigma }^2)& =\prod _{i=1}^n{\left(\frac{1}{2\pi {\sigma }^2}\right)}^{1/2}\exp \left(-\frac{(y_i-{\beta }_0-{\sum }_j{\beta }_j{x}_{i,j}{)}^2}{2{\sigma }^2}\right)\\ & ={\left(\frac{1}{2\pi {\sigma }^2}\right)}^{n/2}\exp \left(-\frac{{\sum }_{i=1}^n(y_i-{\beta }_0-{\sum }_j{\beta }_j{x}_{i,j}{)}^2}{2{\sigma }^2}\right)\\ & ={\left(\frac{1}{2\pi {\sigma }^2}\right)}^{n/2}\exp \left(-\frac{\text{RSS}({\beta }_0,\mathbf{\beta })}{2{\sigma }^2}\right)\\ L(\mathbf{y}\mid {\beta }_0,\mathbf{\beta },{\sigma }^2)& =-\log p=\frac{n}{2}\log (2\pi {\sigma }^2)+\frac{\text{RSS}({\beta }_0,\mathbf{\beta })}{2{\sigma }^2}\end{array}$$

• Step 1 justification ➔ independence turns the joint density into a product; $e^{-a}{e}^{-b}=e^{-a-b}$ collapses the exponentials into one sum.
• Step 2 justification ➔ that sum is the residual sum-of-squares by definition.
• Step 3 justification ➔ in $L$, the first term is free of ${\beta }_0,\mathbf{\beta }$ and the second is $\text{RSS}$ divided by the positive constant $2{\sigma }^2$ ➔ the minimiser is unchanged by that scaling.

Q.E.D. ${\hat{\beta }}_0,\hat{\mathbf{\beta }}$ from least squares are exactly the maximum-likelihood estimates under normal errors. $■$

⚠️ Common Mistakes

• 💡 Claiming LS “assumes normality” ➔ least squares is a loss choice valid with no distributional assumption; normality is what makes it coincide with ML and is what the $p$-values and predictive intervals need.
• 💡 Dividing by $n-1$ for the regression variance ➔ the unbiased divisor is $n-p-1$; $n-1$ is the special case $p=0$ (the mean model).
• 💡 Using ${\hat{\sigma }}_{ML}^2$ for inference ➔ it is biased low ⟹ over-narrow prediction intervals and over-confident tests; use ${\hat{\sigma }}_u^2$. It is the right one inside the information criteria, where the NLL is evaluated at the ML estimates.

🧠 Active Recall

Why does ${\sigma }^2$ not affect which $\mathbf{\beta }$ maximises the likelihood?

• Hint: Look at where $\mathbf{\beta }$ appears in $L$.

Answer

• Short answer: $\mathbf{\beta }$ appears only inside $\text{RSS}$, which enters $L$ divided by the positive constant $2{\sigma }^2$; scaling a function by a positive constant moves its value but not its argmin.
• Why: $L=\frac{n}{2}\log (2\pi {\sigma }^2)+\frac{\text{RSS}}{2{\sigma }^2}$ ➔ the first term is $\mathbf{\beta }$-free, so $\arg {\min }_{\mathbf{\beta }}L=\arg {\min }_{\mathbf{\beta }}\text{RSS}$ for every ${\sigma }^2>0$.

Why is the unbiased error-variance divisor $n-p-1$ rather than $n$?

Answer

• Short answer: Fitting $p+1$ coefficients uses up $p+1$ degrees of freedom — the residuals are shrunk by construction, so dividing by $n$ systematically understates the spread.
• Why: Degrees of freedom ➔ ${\hat{\sigma }}_{ML}^2=\text{RSS}/n$ is biased low; ${\hat{\sigma }}_u^2=\text{RSS}/(n-p-1)$ corrects it, and the gap widens as $p$ approaches $n$.