Logistic Regression

Context: FIT2086_MOC · the binary-target counterpart of Linear Regression (FIT2086) — same linear predictor ${\eta }_i$, squashed into $(0,1)$ · it is the discriminative answer to the curse of dimensionality in Classification and Conditional Class Probabilities · fitted by ML on a Bernoulli target · pruned by penalised likelihood · scored by Classification Evaluation (Confusion Matrix and Metrics), ROC and AUC, Logarithmic Loss · run it ➔ Logistic Regression in R (glm, pROC, step)

Quick Revision

• 🎯 Objective: model the log-odds of $Y=1$ as a linear function of the predictors ➔ $\log \frac{\mathbb{P}(Y_i=1\mid {\mathbf{x}}_i)}{\mathbb{P}(Y_i=0\mid {\mathbf{x}}_i)}={\beta }_0+{\sum }_j{\beta }_j{x}_{i,j}\equiv {\eta }_i$ ➔ invert with the logistic function to get a probability.
• 📦 Core Components: linear predictor ${\eta }_i\in (-\infty ,\infty )$ ➔ odds $e^{{\eta }_i}\in (0,\infty )$ ➔ probability $\frac{1}{1+e^{-{\eta }_i}}\in (0,1)$.
• ⚡ Key Constraint: the negative log-likelihood has no closed-form solution ➔ the estimates must be found numerically; it is convex, so there are no spurious local minima.

📝 How It Works

1. Why the Linear Model Fails on a Binary Target

• The naive attempt ➔ fit $\mathbb{E}[Y_i]={\eta }_i={\beta }_0+{\sum }_{j=1}^p{\beta }_j{x}_{i,j}$ by least squares and read $\mathbb{P}(Y_i=1\mid {\mathbf{x}}_i)\approx {\eta }_i$.
• The fatal flaw ➔ ${\eta }_i$ is unbounded ➔ for extreme feature values it returns $\hat{p}<0$ or $\hat{p}>1$, which are not probabilities.
• The fix ➔ keep the linear predictor, but pass it through a function that bounds it to $(0,1)$. Many such functions exist; logistic regression picks one.

2. The Logistic Function

• Definition ➔ $\mathbb{P}(Y_i=1\mid x_{i,1},\dots ,x_{i,p})=\frac{1}{1+\exp (-{\eta }_i)}$.
• Limits ➔ smoothly $\to 0$ as ${\eta }_i\to -\infty$ and $\to 1$ as ${\eta }_i\to +\infty$; it passes through $\frac{1}{2}$ at ${\eta }_i=0$.
• Shape ➔ an S-curve (sigmoid) — steepest near ${\eta }_i=0$, flat in both tails ⟹ the same unit change in a predictor moves the probability most for individuals near the decision boundary.

3. Odds and Log-Odds

• Odds ➔ $\frac{\mathbb{P}(Y=1)}{\mathbb{P}(Y=0)}=\frac{\mathbb{P}(Y=1)}{1-\mathbb{P}(Y=1)}$ — how many times more likely $Y=1$ is than $Y=0$.
• Worked reading ➔ a coin with $\mathbb{P}(\text{head})=0.75$ has odds $0.75/0.25=3$ for a head and $0.25/0.75=1/3$ for a tail — the two are reciprocals, not symmetric around a centre.
• Log-odds restore symmetry ➔ $\log 3$ and $-\log 3$ ➔ the log scale maps $(0,\infty )$ onto $(-\infty ,\infty )$, which is exactly the range a linear predictor lives in.
• Odds vs probability ➔ odds $>1⟺\mathbb{P}(Y=1)>\frac{1}{2}$; odds $=1⟺$ equally likely; odds $<1⟺Y=0$ more likely.

4. The Model and Its Derivation

• The model ➔ the conditional log-odds are the linear predictor:

$$\log \left(\frac{\mathbb{P}(Y_i=1\mid x_{i,1},\dots ,x_{i,p})}{\mathbb{P}(Y_i=0\mid x_{i,1},\dots ,x_{i,p})}\right)={\beta }_0+\sum _{j=1}^p{\beta }_j{x}_{i,j}\equiv {\eta }_i$$

• Recovering the logistic function ➔ exponentiate, then solve for the probability:

$$\begin{array}{rl}\log \left(\frac{\mathbb{P}(Y_i=1\mid {\mathbf{x}}_i)}{1-\mathbb{P}(Y_i=1\mid {\mathbf{x}}_i)}\right)& ={\eta }_i\\ \frac{\mathbb{P}(Y_i=1\mid {\mathbf{x}}_i)}{1-\mathbb{P}(Y_i=1\mid {\mathbf{x}}_i)}& =\exp ({\eta }_i)\\ \mathbb{P}(Y_i=1\mid {\mathbf{x}}_i)& =\frac{\exp ({\eta }_i)}{1+\exp ({\eta }_i)}=\frac{1}{1+\exp (-{\eta }_i)}\end{array}$$

• Why the two forms are the same ➔ divide numerator and denominator by $\exp ({\eta }_i)$ and use $1/e^a=e^{-a}$ ⟹ “linear log-odds” and “logistic function” are one assumption stated two ways.

5. Fitting by Maximum Likelihood

• Distributional assumption ➔ targets are independent and $Y_i\sim \text{Be}({\theta }_i({\beta }_0,\mathbf{\beta }))$ with ${\theta }_i=\frac{1}{1+\exp (-{\eta }_i)}$ ⟹ $\mathbb{P}(Y_i=1)={\theta }_i$, $\mathbb{P}(Y_i=0)=1-{\theta }_i$.
• Likelihood ➔ the Bernoulli pmf, multiplied over the sample:

$$p(\mathbf{y}\mid {\beta }_0,\mathbf{\beta })=\prod _{i=1}^n{\theta }_i^{y_i}(1-{\theta }_i{)}^{1-y_i}$$

• Negative log-likelihood ➔ take $-\log$, then substitute the logistic form to reach the compact version in ${\eta }_i$:

$$\begin{array}{rl}L(\mathbf{y}\mid {\beta }_0,\mathbf{\beta })& =-\sum _{i=1}^n[y_i\log {\theta }_i+(1-y_i)\log (1-{\theta }_i)]\\ & =\sum _{i=1}^n[-y_i{\eta }_i+\log (1+e^{{\eta }_i})]\end{array}$$

• The estimates ➔ ${\hat{\beta }}_0,\hat{\mathbf{\beta }}$ are the minimisers of $L$ ➔ the ML estimates, exactly as in Maximum Likelihood Estimation.
• No closed form ➔ unlike least squares there is no formula ⟹ solved numerically (gradient descent / Newton-type optimisers).
• Convexity is the safety net ➔ $L$ is convex in the coefficients, so a numerical search cannot get stuck in a spurious local minimum, assuming a finite ML estimate exists.
• Cost ➔ grows with both $n$ and $p$, but is $O(n{p}^2)$ per iteration rather than exponential ➔ this is the escape from the curse of dimensionality.

6. Interpreting the Coefficients

• Intercept ${\beta }_0$ ➔ the log-odds of $Y=1$ when every predictor is zero ($x_{i,1}=\cdots =x_{i,p}=0$) — usually an extrapolation with no real-world individual behind it.
• Coefficient ${\beta }_j$ ➔ the increase in log-odds per one-unit increase in $x_j$, holding the others fixed ⟹ a multiplicative factor $e^{{\beta }_j}$ on the odds, not an additive change in probability.
• Sign reading ➔ ${\eta }_i>0\Rightarrow e^{{\eta }_i}>1\Rightarrow Y=1$ more likely; ${\eta }_i<0\Rightarrow e^{{\eta }_i}<1\Rightarrow Y=0$ more likely.
• Goodness-of-fit ➔ the minimised $L(\mathbf{y}\mid {\hat{\beta }}_0,\hat{\mathbf{\beta }})$ — smaller is better; compare it against the intercept-only model $L(\mathbf{y}\mid {\hat{\beta }}_0)$, whose difference plays the role $\text{RSS}\to \text{TSS}$ plays for $R^2$. The intercept-only model assumes $\mathbb{P}(Y=1)$ is the same for everybody.
• Deviance (Studio 7) ➔ R reports $2L$: null deviance $=2L(\mathbf{y}\mid {\hat{\beta }}_0)$, residual deviance $=2L(\mathbf{y}\mid {\hat{\beta }}_0,\hat{\mathbf{\beta }})$; the gap measures improvement over the intercept-only model.

7. Predicting, Deciding and Pruning

• Predict ➔ compute $\hat{\eta }={\hat{\beta }}_0+{\sum }_{j=1}^p{\hat{\beta }}_j{x}_j^{\prime }$ for new features, then $\mathbb{P}(Y^{\prime }=1\mid {\mathbf{x}}^{\prime })=\frac{1}{1+\exp (-\hat{\eta })}$.
• Decision rule ➔ pick the class maximising that probability ⟹ equivalently predict $1$ iff $\mathbb{P}(Y=1\mid {\mathbf{x}}^{\prime })>\frac{1}{2}$, iff the odds exceed $1$, iff $\hat{\eta }>0$ — three views of one rule ➔ generalised to any threshold in ROC and AUC.
• Categorical and non-linear predictors ➔ handled exactly as in linear regression — $K-1$ indicator columns for a $K$-category predictor when an intercept is present, plus $\log$ and polynomial terms ➔ Predictor Transformations (Indicators, Polynomials, Interactions).
• Which predictors ➔ test $H_0:{\beta }_j=0$ vs $H_A:{\beta }_j\ne 0$ (smaller $p$-value ⟹ stronger predictor), or score models by penalised likelihood $L(\mathbf{y}\mid {\hat{\beta }}_0,\hat{\mathbf{\beta }})+k{\alpha }_n$ with ${\alpha }_n=1$ (AIC), ${\alpha }_n=\frac{3}{2}$ (KIC), ${\alpha }_n=\frac{1}{2}\log n$ (BIC), then search by forward/backward selection ➔ Model Selection and Information Criteria (AIC, BIC).
• Interactions ➔ add $x_j{x}_k$ ⟹ the change in log-odds per unit of $x_j$ now depends on $x_k$ (and vice versa); without it the change is the same whatever the other predictors are.
• The boundary is linear ➔ the set where $\mathbb{P}(Y=1\mid x_1,x_2)=\frac{1}{2}$ is exactly $\eta =0$, a line in two features and a $p$-dimensional hyperplane in general ⟹ classes that are not linearly separable in the feature space cannot be separated by logistic regression without transformed predictors.

🔬 Model

• Declared spaces ➔ $\mathbf{y}\in \{0,1{\}}^n$, design matrix $X\in {\mathbb{R}}^{n\times p}$ (plus an intercept column), $\mathbf{\beta }\in {\mathbb{R}}^p$, ${\beta }_0\in \mathbb{R}$, ${\theta }_i\in (0,1)$.
• Link ➔ ${\eta }_i={\beta }_0+{\sum }_{j=1}^p{\beta }_j{x}_{i,j}$, ${\theta }_i=\frac{1}{1+\exp (-{\eta }_i)}$, $Y_i\mid {\mathbf{x}}_i\sim \text{Be}({\theta }_i)$ independently.
• Objective (minimise) ➔

$$({\hat{\beta }}_0,\hat{\mathbf{\beta }})=\arg \underset{{\beta }_0,\mathbf{\beta }}{\min }\sum _{i=1}^n[-y_i{\eta }_i+\log (1+e^{{\eta }_i})]$$

• Contrast with Linear Regression (FIT2086) ➔ same ${\eta }_i$; the Gaussian mean model becomes a Bernoulli success probability, and the closed-form least-squares solution becomes a numerical convex optimisation ➔ the ML framing of Least Squares as Maximum Likelihood is what makes the swap legal.

⚖️ Core Decision Matrix

Model for a binary target	What is linear in $\mathbf{x}$	Fitted by	Output range	Where it breaks
Linear probability model	the probability itself	least squares, closed form	$(-\infty ,\infty )$ ➔ invalid	returns $\hat{p}<0$ or $\hat{p}>1$ at extreme $\mathbf{x}$
Logistic regression	the log-odds	numerical ML, convex	$(0,1)$ ➔ always valid	decision boundary forced linear; multi-class awkward

When It Flips: over a narrow band of $\mathbf{x}$ where all fitted probabilities sit near $0.5$, the logistic curve is close to straight and the two models agree ➔ the linear probability model only fails once the data reach into the tails, which real feature ranges almost always do.

📊 Exam Execution Trace & Applied Exercises

Applied Exercise 1 — from log-odds to odds to probability

Problem: a fitted model gives $\text{LogOdds}(Y)=0.5+3x$. At $x=0.2$, report the odds and the probability that $Y=1$.

$$\begin{array}{rl}\eta & =0.5+3(0.2)=1.1\\ \text{odds}(Y=1)& =e^{1.1}\approx 3.00\\ \mathbb{P}(Y=1)& =\frac{1}{1+e^{-1.1}}=\frac{1}{1+0.3329}\approx 0.75\end{array}$$

Final Extracted Output: $Y=1$ is about three times as likely as $Y=0$; probability $\approx 0.75$ ⟹ predict class $1$ (odds $>1$).

Applied Exercise 2 — fit interpretation and prediction

Problem: high blood pressure is defined as ${\text{HighBP}}_i=I({\text{BP}}_i>120)$; ML fitting on Age gives $\text{logOdds}({\text{HighBP}}_i)=-33.68+0.656{\text{Age}}_i$. Interpret both coefficients and predict for a $50$-year-old.

$$\begin{array}{rl}\eta & =-33.68+0.656(50)=-33.68+32.80=-0.88\\ \text{odds}& =e^{-0.88}\approx 0.41\\ \mathbb{P}(\text{HighBP}=1\mid \text{Age}=50)& =\frac{0.41}{1+0.41}\approx 0.29\end{array}$$

Final Extracted Output: at $\text{Age}=0$ the log-odds are $-33.68$ (a meaningless extrapolation); each extra year adds $0.656$ to the log-odds, i.e. multiplies the odds by $e^{0.656}\approx 1.93$. A $50$-year-old has $\approx 29\%$ probability ⟹ predict $\text{HighBP}=0$. The model crosses $\frac{1}{2}$ at $\text{Age}=33.68/0.656\approx 51.3$ years.

Applied Exercise 3 — reading a pruned model (Studio 7)

Problem: BIC stepwise on $100$ binary SNP predictors ($n=200$) keeps $\hat{\eta }=-0.112-1.134\text{SNP12}+1.460\text{SNP56}$ (indicators $=1$ when mutated). Give the disease probability with neither, only SNP12, and only SNP56 mutated.

$$\begin{array}{rlrl}\text{neither: }\hat{\eta }& =-0.112& \Rightarrow \hat{p}& =\frac{1}{1+e^{0.112}}\approx 0.47\\ \text{SNP12: }\hat{\eta }& =-0.112-1.134=-1.246& \Rightarrow \hat{p}& \approx 0.22\\ \text{SNP56: }\hat{\eta }& =-0.112+1.460=1.348& \Rightarrow \hat{p}& \approx 0.79\end{array}$$

Final Extracted Output: a SNP12 mutation lowers the log-odds by $1.134$ (odds $\times e^{-1.134}\approx 0.32$); a SNP56 mutation raises them by $1.460$ (odds $\times e^{1.460}\approx 4.3$).

⚠️ Common Mistakes

• 💡 Reading ${\beta }_j$ as a change in probability ➔ it is a change in log-odds; the same ${\beta }_j$ moves the probability a lot near $\eta =0$ and almost nothing in the tails, so no single “percentage point” answer exists.
• 💡 Exponentiating the wrong thing ➔ $e^{\eta }$ is the odds, not the probability; the probability needs the extra step $\frac{\text{odds}}{1+\text{odds}}$ or $\frac{1}{1+e^{-\eta }}$.
• 💡 Comparing likelihood numbers across packages ➔ some report maximised log-likelihood, some the negative, some twice either ➔ a “bigger is better” habit inverts the verdict.
• 💡 Forgetting the boundary is linear ➔ a poor fit on visibly curved class structure is a model-form failure, not an optimisation failure ➔ transform the predictors ($x^2$, $x^3$, $\log x$) or switch to a tree / forest, which is non-linear natively · (Studio 12 exam phrasing) off a scatter: “a straight line would not separate the classes, a simple curve would” ⟹ logistic regression is inappropriate.

🧠 Active Recall

Why model the log-odds rather than the probability directly?

• Hint: Compare the range of each side of the equation.

Answer

• Short answer: The linear predictor ${\eta }_i$ ranges over all of $\mathbb{R}$, and so do the log-odds — but a probability only ranges over $(0,1)$, so equating ${\eta }_i$ to a probability produces impossible values.
• Why: Range matching ➔ probability $(0,1)\stackrel{\text{odds}}{\to }(0,\infty )\stackrel{\log }{\to }(-\infty ,\infty )$ ➔ the log-odds is the natural scale for an unbounded linear function, and the logistic function is just this chain run backwards.

Least squares has a formula; logistic regression does not. Why is that not a practical disaster?

• Hint: What shape is the objective surface?

Answer

• Short answer: The negative log-likelihood is convex, so any descent method reaches the global minimum — there are no spurious local minima to trap it.
• Why: Convex objective ➔ $L={\sum }_i[-y_i{\eta }_i+\log (1+e^{{\eta }_i})]$ has a single basin ⟹ the estimate is well defined even without a formula, provided a finite ML estimate exists (it fails on perfectly separable data, where coefficients run to infinity).

A model gives $\hat{\eta }=0$ for an individual. State the odds, the probability, and the predicted class.

Answer

• Short answer: Odds $=e^0=1$, probability $=\frac{1}{1+e^0}=\frac{1}{2}$, and the classes are exactly tied — the individual sits on the decision boundary.
• Why: The three equivalent rules ➔ $\hat{\eta }>0⟺\text{odds}>1⟺\hat{p}>\frac{1}{2}$; at $\hat{\eta }=0$ all three are at their pivot, which is precisely the hyperplane $\mathbb{P}(Y=1\mid \mathbf{x})=\frac{1}{2}$.

How is "which predictors belong in the model" answered here, given there is no $R^2$?

Answer

• Short answer: Identically to linear regression, because logistic regression is fitted by ML and therefore has a likelihood — test $H_0:{\beta }_j=0$, or score models by $L+k{\alpha }_n$ and search by stepwise selection.
• Why: Penalised likelihood transfers ➔ the minimised NLL always falls as predictors are added (exactly as $R^2$ always rises), so a complexity charge ${\alpha }_n\in \{1,\frac{3}{2},\frac{1}{2}\log n\}$ for AIC/KIC/BIC is what makes two models comparable ➔ Model Selection and Information Criteria (AIC, BIC).