FIT2086 Unit Cheatsheet

Context: FIT2086_MOC · the paper-exam half (50%) of W1–W11 in one re-read, in the order the W12 revision lecture runs · R syntax lives in R Toolkit (Cheatsheet) · links for depth only. Scope: hand-worked exam material only — R implementation (A1–A3) lives in the R Toolkit. Read order: ⛔ = precondition / trap where marks die · 📄 = printed on the formula sheet (recognise, don’t memorise) · 🧮 Blueprint = derivation to reproduce · ⚙️ Procedure = hand computation to run · Priority: §1–§6 are L1–L9 (revised in W12 and examined in the sample paper) · §7–§8 are lower signal.

Quick Revision

• 🎯 Objective: name the model ➔ write the formula with its precondition ➔ compute ➔ close with a sentence about the population, in the question’s units.
• ⚡ Key Constraint: marks sit in the stated assumption (iid, known $\sigma$, independence, sidedness) and the closing interpretation — a bare number earns part marks only.

0️⃣ Exam Anatomy (Studio 12 sample paper)

Section	Examined act	Crib	Drill in
Short answer	2–3 sentence definition	recipe below	folded answers below
MLE	likelihood ➔ NLL ➔ $dL/d\theta =0$ (sample: $\mathrm{Exp}(\beta )$)	§2️⃣	Maximum Likelihood Estimation Practice 1
Random variables	$E$/$V$ of sums and products of independent RVs	§1️⃣	Expectations and Covariance (FIT2086)
CI and $p$-values	known-$\sigma$ $z$ interval $+$ $z$-test by table $+$ summary sentence	§3️⃣	Tests for Normal Means (z-test and t-test)
Regression	LS objective, $K-1$ indicators, read ${\beta }_j$, predict, $R^2$	§4️⃣	Linear Regression (FIT2086)
Classification	conditional from a joint table, odds, predictor verdict	§5️⃣	Classification and Conditional Class Probabilities
Logistic regression	read ${\beta }_j$ on log-odds, $\eta \to p\to$ class at $\frac{1}{2}$, non-linearity fix	§5️⃣	Logistic Regression
Bias & variance	prove $\bar{Y}$ unbiased, $V[\bar{Y}]={\sigma }^2/n$, consistent	§2️⃣	Estimator Quality (Bias, Variance, MSE)
Machine learning	read an R tree print-out · tree vs linear · kNN vote off a scatter · is the boundary linear?	§6️⃣	tree print-out Practice 1, k-Nearest Neighbours

📄 Supplied on the formula sheet	Recall cold
$E[X]$ · marginal and conditional formulas	every pmf/pdf, family mean and variance
$\frac{d}{dx}{x}^k$, $\frac{d}{dx}\log x$, chain rule	MLE results, $V[\bar{Y}]$
known-$\sigma$ CI and $z$-test (one mean, a difference)	regression and logistic formulas, $R^2$
$P(Z<-|z|)$ table	IC penalties, every ML algorithm

• Scope signal ➔ the W12 lecture and the sample paper both cover L1–L9 only · ⛔ neither excludes W10–11 — the paper is “roughly commensurate”, not a syllabus.
• Table lookup ➔ rows step by $0.093$ ⟹ take the nearest $|z|$ ($1.763\to 1.767$, $0.0386$), double it for a two-sided $p$.
• Short-answer recipe ➔ sentence 1 says what it is; sentences 2–3 give one or two properties — shows you can identify it and know something about it.

Model short answers (Studio 12)

• Bias / variance of an estimator ➔ bias is $E[\hat{\theta }]-\theta$, the average amount the estimator under- or over-estimates; zero ⟹ unbiased. Variance is the average squared deviation of $\hat{\theta }$ from its own mean — how much the estimate would move on a fresh sample.
• $p$-value ➔ measures evidence against $H_0$ in a hypothesis test: the probability of a test statistic as or more extreme than the one observed, by chance alone, if $H_0$ were true.
• Accuracy / sensitivity / specificity ➔ percentage of individuals classified correctly · proportion of actual $1$s classified as $1$ · proportion of actual $0$s classified as $0$.
• Decision tree ➔ a supervised learner that splits the data into disjoint leaves by binary questions on the attributes; each leaf holds its own simple model used to predict.
• Penalized regression ➔ estimates linear/logistic coefficients by minimising goodness-of-fit (e.g. RSS) plus a penalty on coefficient size; the penalty curbs overfitting.
• Random variable ➔ takes one value from a set $\mathcal{X}$ with frequencies set by its distribution; many realisations give observed frequencies close to the theoretical ones.

1️⃣ Modelling, Description & Random Variables (W1–2)

• The modelling pipeline ➔ population (what we want) → sampling → sample $\mathbf{y}$ → inference (fit a model with unknown parameters $\theta$, particular → general) → model checking (goodness-of-fit) · a model is never correct, only more or less useful.
• Three sources of randomness ➔ measurement error · unmeasured factors · random sampling (which items entered the sample) · ⛔ “random” models sampling variability, not an unpredictable reality.
• Descriptive vs inferential ➔ a statistic is any function $s(\mathbf{y})$ and describes only the sample · inference generalises to the population and needs a model · a good statistic often becomes an estimator ($\bar{y}$ for $\mu$).

Centre / spread	Formula	Robust to outliers?
Mean	$\bar{y}=\frac{1}{n}{\sum }_i{y}_i$	❌
Median	$Q(\mathbf{y},50)$	✅
Range	$\max -\min$	❌
Variance / SD	$s^2=\frac{1}{n}{\sum }_i(y_i-\bar{y}{)}^2$ · $s=\sqrt{s^2}$ (data’s units)	❌
IQR	$Q_3-Q_1=Q(\mathbf{y},75)-Q(\mathbf{y},25)$	✅

• Skew from mean − median ➔ mean $>$ median ⟹ right tail · $<$ ⟹ left tail · $\approx$ ⟹ symmetric · boxplot $=$ five-number summary · outliers beyond $Q_3+1.5\text{IQR}$ or below $Q_1-1.5\text{IQR}$.
• Association by type pair ➔ cont × cont: scatter $+$ Pearson $r\in [-1,1]$ · cat × cat: side-by-side bars / contingency table · cat × num: side-by-side boxplots · ⛔ $r$ sees linear association only ($y=x^2$ gives $r\approx 0$), and correlation $\ne$ causation.
• Sum rule 📄 ➔ $P(X=x)={\sum }_{y}P(X=x,Y=y)$ · conditional 📄 $P(X=x\mid Y=y)=\frac{P(X=x,Y=y)}{P(Y=y)}$ · ⛔ divide by the marginal of the conditioning variable.
• CDF / survival ➔ $P(X\le x)={\sum }_{x^{\prime }\le x}P(X=x^{\prime })$ · $P(X>x)=1-P(X\le x)$ · continuous: $P(X=x)=0$.
• Median ➔ continuous: solve $P(X\le x)=0.5$ · discrete: smallest $x$ with $P(X\le x)\ge 0.5$ and $P(X\le x-1)<0.5$.
• iid likelihood ➔ $p(\mathbf{y}\mid \theta )={\prod }_{i}p(y_i\mid \theta )$ · ⛔ the product needs independence — say so in line one of any MLE.
• Shared maths ➔ Random Variable · Conditional Probability · Bayes’ Theorem · Expectation · Variance and Standard Deviation (FIT1058 cluster).
• Expectation ➔ $E[f(X)]={\sum }_{x}p(x)f(x)$ (integral if continuous) · $V[X]=E[X^2]-E[X{]}^2$ ⟺ $E[X^2]=V[X]+E[X{]}^2$.

Identity	Holds
$E[f(X)+g(Y)]=E[f(X)]+E[g(Y)]$	always
$E[cf(X)]=cE[f(X)]$ · $V[cf(X)+d]=c^{2}V[f(X)]$	always
$E[f(X)g(Y)]=E[f(X)]E[g(Y)]$	⛔ independent only
$V[f(X)+g(Y)]=V[f(X)]+V[g(Y)]$	⛔ independent only
$V[cX-dY]=c^{2}V[X]+d^{2}V[Y]$ — variances add for a difference	⛔ independent only
independent ⟹ $\mathrm{cov}=0$	one way only: $\mathrm{corr}=0\nRightarrow$ independent

• Taylor ➔ $E[f(X)]\approx f(\mu )+\frac{{\sigma }^2}{2}{f}^{\prime \prime }(\mu )$ · $V[f(X)]\approx {\sigma }^2{f}^{\prime }(\mu {)}^2$ ➔ Taylor Approximation of Expectations.

• Parametric family ➔ $p(x\mid \theta )$ generates every probability from $k\ll |\mathcal{X}|$ parameters; $E$, $V$, cdf and quantiles are all functions of $\theta$ · ⛔ state the parameterisation ($U(k)$ vs $U(a,b)$) before quoting a mean ➔ Random Variables and Probability Distributions (FIT2086).

• Pick the family by the support ➔ $\{0,1\}$ Bernoulli · $\{0,\dots ,n\}$ successes in $n$ trials binomial · unbounded counts in an interval Poisson · equally likely outcomes uniform · $\mathbb{R}$ Gaussian · ⛔ the reason is always support / number of outcomes, never “looks bell-shaped”.

• Variance–mean fingerprint ➔ Poisson $V=E$ · binomial $V<E$, capped, maximal at $\theta =\frac{1}{2}$ · Gaussian leaves $\mu$, ${\sigma }^2$ free.

Family	Support	$p(x)$	$E[X]$	$V[X]$	ML estimate
Gaussian $N(\mu ,{\sigma }^2)$	$\mathbb{R}$	$\frac{1}{\sqrt{2\pi {\sigma }^2}}{e}^{-(x-\mu {)}^2/(2{\sigma }^2)}$	$\mu$	${\sigma }^2$	$\bar{y}$ · $\frac{1}{n}\sum (y_i-\bar{y}{)}^2$
$Be(\theta )$	$\{0,1\}$	${\theta }^x(1-\theta {)}^{1-x}$	$\theta$	$\theta (1-\theta )$	$m/n$
Binomial $Bin(\theta ,n)$	$\{0,\dots ,n\}$	$\left(\genfrac{}{}{0ex}{}{n}{x}\right){\theta }^x(1-\theta {)}^{n-x}$	$n\theta$	$n\theta (1-\theta )$	—
Poisson $Poi(\lambda )$	$\{0,1,\dots \}$	$\frac{{\lambda }^x{e}^{-\lambda }}{x!}$	$\lambda$	$\lambda$	$\bar{y}$
Uniform $U(k)$	$\{1,\dots ,k\}$	$\frac{1}{k}$	$\frac{k+1}{2}$	$\frac{k^2-1}{12}$	—
$U(a,b)$	$[a,b]$	$\frac{1}{b-a}$	$\frac{a+b}{2}$	$\frac{(b-a{)}^2}{12}$	—
$\mathrm{Exp}(\beta )$	$y\ge 0$	$\beta e^{-\beta y}$	—	—	$n/{\sum }_i{y}_i=1/\bar{y}$

• Additivity ⛔ independent ➔ $Poi({\lambda }_1)+Poi({\lambda }_2)=Poi({\lambda }_1+{\lambda }_2)$ · Gaussians add means and variances · standardise $Z=\frac{X-\mu }{\sigma }$.
• WLLN ➔ $\bar{Y}\to E[Y]$ as $n\to \infty$ · ⛔ only if $E[Y]$ exists.

2️⃣ Estimation (W3)

🧮 Blueprint — MLE

① $p(\mathbf{y}\mid \theta )={\prod }_{i}p(y_i\mid \theta )$ — state iid ② $L(\theta )=-\log p(\mathbf{y}\mid \theta )$ using $\log ab=\log a+\log b$, $\log a^b=b\log a$, $\log e^a=a$ ③ $\frac{dL}{d\theta }=0$ ④ solve for $\hat{\theta }$. Calculus not on the sheet: $\frac{d}{dx}{e}^x=e^x$ · product rule $(fg{)}^{\prime }=g{f}^{\prime }+f{g}^{\prime }$ · partial $\partial /\partial {\theta }_j$ freezes every other parameter (one equation per parameter) · ⛔ $\log (a+b)\ne \log a+\log b$ · ⛔ $\frac{d}{dx}\log g(x)=\frac{g^{\prime }(x)}{g(x)}$ — the inner derivative is the usual omission · log is monotone ⟹ same maximiser, easier sum.

• Sampling distribution ➔ $\hat{\theta }(Y_1,\dots ,Y_n)$ is a function of RVs, so it is an RV — its distribution is how $\hat{\theta }$ varies over repeated samples · uses: CIs, tests, comparing estimators.
• Assumption strength ➔ iid with mean $\mu$, variance ${\sigma }^2$ gives $E[\bar{Y}]=\mu$, $V[\bar{Y}]={\sigma }^2/n$ but no shape · iid $N(\mu ,{\sigma }^2)$ gives $\bar{Y}\sim N(\mu ,{\sigma }^2/n)$ — the shape is what licenses probability statements · ⛔ $V[\bar{Y}]$ is $n\times$ smaller than $V[Y]$.
• Plug-in ➔ probability statements use $p(y\mid \hat{\theta })$ ➔ Plug-in Prediction and Held-Out Evaluation.
• Quality ➔ bias $b_{\theta }=E[\hat{\theta }]-\theta$ · variance $V[\hat{\theta }]$ · $\mathrm{MSE}=b^2+\mathrm{Var}$ · more efficient $=$ smaller MSE (smaller variance among unbiased).

🧮 Blueprint — $\bar{Y}$ is unbiased and consistent

① $E[\bar{Y}]=\frac{1}{n}\sum E[Y_i]=\mu$ (linearity) ⟹ unbiased ② $V[\bar{Y}]=\frac{1}{n^2}\sum V[Y_i]=\frac{{\sigma }^2}{n}$ ⛔ independence ③ $\mathrm{MSE}={\sigma }^2/n\to 0$ ⟹ consistent ($b\to 0$ and $\mathrm{Var}\to 0$).

• ML variance is biased ➔ $E[{\hat{\sigma }}_{ML}^2]=\frac{n-1}{n}{\sigma }^2$ ⟹ ${\hat{\sigma }}_u^2$ divides by $n-1$ ➔ Estimator Quality (Bias, Variance, MSE).

🧮 Blueprint — shrinkage estimator $\hat{\mu }(c)=\frac{n}{n+c}\bar{Y}$, $c>0$ (Studio 8)

① $E=\frac{n\mu }{n+c}$ ⟹ $b=-\frac{c\mu }{n+c}$ ② $V={\left(\frac{n}{n+c}\right)}^2\frac{{\sigma }^2}{n}=\frac{n{\sigma }^2}{(n+c{)}^2}$ ⛔ square the multiplier ③ $\mathrm{MSE}=\frac{c^2{\mu }^2+n{\sigma }^2}{(n+c{)}^2}$ ④ beats $\bar{Y}$ iff ${\mu }^2<{\sigma }^2\frac{2n+c}{nc}$ — only near the target $0$ ⑤ $b,V\to 0$ as $n\to \infty$ ⟹ still consistent. It is ridge: $\hat{\mu }(c)=\arg {\min }_{\mu }\{\frac{1}{2}{\sum }_i(y_i-\mu {)}^2+\frac{c}{2}{\mu }^2\}$ — $c$ plays $\lambda$ · ⛔ “biased ⟹ worse” is false: bias is a cost traded for variance.

3️⃣ CLT, Intervals & Tests (W4–5)

• CLT ➔ ${\sum }_i{Y}_i\stackrel{d}{\to }N(n\mu ,n{\sigma }^2)$ ⟹ $\bar{Y}\stackrel{d}{\to }N(\mu ,{\sigma }^2/n)$ · ⛔ finite $\mu ,{\sigma }^2$ and large $n$ — any family ➔ Central Limit Theorem.
• CI meaning ➔ $P(\theta \in ({\hat{\theta }}^{-},{\hat{\theta }}^{+}))=1-\alpha$ over repeated samples · ⛔ the procedure covers — never “this interval has a $95\%$ chance”.

Situation	Interval	⛔ Precondition
one mean, $\sigma$ known 📄	$\hat{\mu }\pm z_{\alpha /2}\frac{\sigma }{\sqrt{n}}$	$\sigma$ known
one mean, $\sigma$ unknown, $n<30$	$\hat{\mu }\pm t_{\alpha /2,n-1}\frac{{\hat{\sigma }}_u}{\sqrt{n}}$ — wider, $t>z$ always ➔ Student-t Distribution	normal population
one mean, $\sigma$ unknown, $n>30$	the $z$ form with $\hat{\sigma }$ plugged in	any population, by the CLT
difference of means 📄	${\hat{\mu }}_A-{\hat{\mu }}_B\pm z_{\alpha /2}\sqrt{\frac{{\hat{\sigma }}_A^2}{n_A}+\frac{{\hat{\sigma }}_B^2}{n_B}}$	approximate, improves with $n$ · contains $0$ ⟹ no clear difference
proportion	$\hat{\theta }\pm z_{\alpha /2}\sqrt{\hat{\theta }(1-\hat{\theta })/n}$ ➔ Tests for Bernoulli Populations	CLT

• Why $t$, not plug-in $z$ ➔ coverage simulation: the known-$\sigma$ $z$ interval covers $95\%$ at every $n$; plugging ${\hat{\sigma }}_u$ into $z$ is too narrow and undercovers at small $n$; the $t$ interval covers $95\%$ at every $n$.
• Critical values ➔ $z_{0.025}=1.96$ · $z_{0.005}=2.576$ · $z_{0.1}=1.281$ ➔ Confidence Intervals, Tests for Normal Means (z-test and t-test).

⚙️ Procedure — $z$-test by table ➔ Hypothesis Testing

① state $H_0$ vs $H_A$ — sidedness from the wording ② $z=\frac{\hat{\mu }-{\mu }_0}{\sigma /\sqrt{n}}$ 📄 ③ $p$ 📄: two-sided $2P(Z<-|z|)$ · $H_A:\mu >{\mu }_0$ ⟹ $1-P(Z<z)$ · $H_A:\mu <{\mu }_0$ ⟹ $P(Z<z)$ ④ grade ⑤ one sentence about the population, in the question’s units.

• Grading ➔ $p>0.05$ weak/none · $0.01$–$0.05$ moderate · $<0.01$ strong · report “1 in $1/p$ by chance” · only if the question fixes $\alpha$: reject iff $p\le \alpha$.
• Never proves $H_0$ ⛔ ➔ large $p$ $=$ data not incompatible with $H_0$ · small $n$ $+$ large effect $+$ borderline $p$ ⟹ get more data.
• CI ⟷ test ➔ ${\mu }_0$ inside the $95\%$ CI ⟺ two-sided $p>0.05$, on the same standard error.

4️⃣ Regression & Complexity Control (W6, W8)

• Model ➔ $E[Y_i]={\beta }_0+{\sum }_j{\beta }_j{x}_{i,j}$ · ${\beta }_j$ $=$ change in $E[Y]$ per unit $x_j$, holding the others constant · ${\beta }_0$ $=$ $E[Y]$ at all-zero predictors.
• Least squares ➔ minimise $\text{RSS}={\sum }_i{e}_i^2$, $e_i=y_i-{\beta }_0-{\sum }_j{\beta }_j{x}_{i,j}$ · ⛔ $p<n$ · Gaussian errors ⟹ LS $=$ ML ➔ Least Squares as Maximum Likelihood.
• $R^2=1-\frac{\text{RSS}}{\text{TSS}}$ ➔ TSS $=$ RSS of the mean model · $1$ ⟺ $\text{RSS}=0$ · unit-free · $0.78$ reads “predictors explain $\approx 78\%$ of the variation in $Y$” · ⛔ rises with every predictor ⟹ never use it to select.
• Building columns ➔ $K$ categories ⟹ $K-1$ indicators, omitted level $=$ reference · polynomial $x,x^2,\dots ,x^q$ · interaction $x_j{x}_k$ · all still linear in $\mathbf{\beta }$ ➔ Predictor Transformations (Indicators, Polynomials, Interactions).

Diagnosis	Training error	Held-out error	Cause
Underfit	high	high	important predictors omitted ⟹ bias
Overfit	low	high — the gap	spurious predictors learn noise ⟹ variance

• Decomposition ➔ $\text{MSE}={\text{bias}}^2+\text{variance}$ ($+$ irreducible ${\sigma }^2$ for a new $Y$).

Complexity control	Rule (NLL scale $L$)	Behaviour
Bonferroni	threshold $\alpha /p$	$p$ tests at $\alpha$ expect $\alpha p$ false positives
AIC	$L+k$	overfits
KIC	$L+\frac{3}{2}k$	—
BIC	$L+\frac{k}{2}\log n$	underfits
RIC	$L+k\log p$	—
Ridge	$\arg \min \{\text{RSS}+\lambda {\sum }_j{\beta }_j^2\}$	shrinks, never zeros
Lasso	$\arg \min \{\text{RSS}+\lambda {\sum }_j|{\beta }_j|\}$	zeros coefficients

• Forward stepwise ➔ empty model ➔ add the predictor cutting the IC most ➔ stop when none improves · backward starts full · ⛔ unstable: a small data change gives a big model change.
• Penalty settings ➔ $\lambda \uparrow$ ⟹ more shrinkage, less flexible · intercept unpenalised (the baseline) · ⛔ standardise predictors first.

⚙️ Procedure — choose $\lambda$ (or any tuning value) by $K$-fold CV

① grid of candidate values ② per value: fit on $K-1$ folds, predict the held-out fold, accumulate the error ③ repeat $m$ times and average ④ pick the smallest error ⑤ refit on all the data.

5️⃣ Classification & Logistic Regression (W7)

• Direct approach ➔ $P(y\mid \mathbf{x})=\frac{P(y,\mathbf{x})}{P(\mathbf{x})}$ off a joint table · ⛔ $2^{p+1}$ joint cells ⟹ curse of dimensionality.
• Odds from a joint table ➔ $\frac{P(Y=1,x)}{P(Y=0,x)}$ — the marginal $P(x)$ cancels.

⚙️ Procedure — logistic prediction

① $\eta ={\beta }_0+{\sum }_j{\beta }_j{x}_j$ (the log-odds) ② odds $=e^{\eta }$ ③ $P(Y=1)=\frac{1}{1+e^{-\eta }}$ ④ predict $1$ iff $P>\frac{1}{2}$ iff $\eta >0$.

• Reading ${\beta }_j$ ➔ change in log-odds per unit $x_j$ ⟺ odds multiplied by $e^{{\beta }_j}$.
• Fit ➔ ML, NLL ${\sum }_i[-y_i{\eta }_i+\log (1+e^{{\eta }_i})]$ · convex, no closed form.
• Boundary is linear ⛔ ➔ a line/hyperplane ⟹ curved class structure needs transformed predictors or a tree/forest (§6️⃣).

Metric	Formula	Reads
Classification accuracy	fraction correct	compare with the class-frequency baseline
Sensitivity	$\frac{TP}{TP+FN}$	actual $1$s caught
Specificity	$\frac{TN}{TN+FP}$	actual $0$s caught
AUC	fraction of positive–negative pairs ranked correctly (ties $\frac{1}{2}$)	$1$ perfect · $\frac{1}{2}$ random
Log-loss	$-\sum \log \hat{P}(\text{true class})$	punishes confident errors

6️⃣ Trees & Nearest Neighbours (W9)

• Machine Learning, the statistician’s view ➔ algorithmic, flexible, usually non-linear, few assumptions · the fitted “model” is often uninterpretable ⟹ the focus is prediction.
• Tree ➔ binary splits ⟹ $L$ disjoint leaves, each with its own model (Bernoulli $n_1/n$ or Normal mean) · complexity $=L$.
• Learning ➔ leaf NLL $-n_1\log \frac{n_1}{n}-n_0\log \frac{n_0}{n}$ · grow greedily ➔ prune back ➔ CV over $L$ ➔ Decision Tree Learning (Likelihood Splits, Pruning, CV).

⚙️ Procedure — read an R tree print-out

① columns node), split, n, deviance, yval; * marks a leaf ② predict: follow the child conditions from the root to a leaf, read its yval ③ max-leaf question: find the largest yval*, then read its path back to the root.

• kNN ➔ standardise ➔ Euclidean distance ➔ $k$ nearest ➔ vote / average / kernel-weighted average · $k$, distance and kernel chosen by LOO CV.

Model	Shape it can learn	Complexity knob	Weakness
Linear / logistic	linear in $\mathbf{\beta }$; hyperplane boundary	predictors (IC, stepwise) or $\lambda$	curvature only via transformed columns
Tree	non-linear relationships and interactions natively	$L$ leaves, by CV	low bias but high variance
Random Forest	as a tree, averaged over bootstrap samples with random candidate variables per split	—	stable, but loses interpretability
kNN	— (check the scatter: is the boundary linear?)	$k$, distance, kernel by LOO CV	⛔ needs standardised predictors

7️⃣ Simulation & Resampling (W10)

• Monte Carlo ➔ $P(\mathbf{X}\in Z)\approx \frac{1}{m}{\sum }_{i}I({\mathbf{X}}^{(i)}\in Z)$ · resolution $1/m$ ➔ Monte Carlo Simulation (Empirical Probabilities) · LCG $X_{n+1}=(a{X}_n+c)\mathrm{mod}m$ ➔ Pseudo-Random Number Generators.
• Comparing estimators by simulation ➔ loop: generate from a chosen $\theta$ → estimate → store · bias $=\overline{\hat{\theta }}-\theta$, variance of the stored values, $\mathrm{MSE}=b^2+\mathrm{Var}$, $\text{RelMSE}=\frac{{\mathrm{MSE}}_1}{{\mathrm{MSE}}_2}$ ($<1$ ⟹ first wins).
• Mean vs median ➔ both unbiased on normal data; the mean is more efficient on clean data ($\text{RelMSE}\approx 0.73$ at $n=10$) · a few contaminated points flip it to the median · the flip is driven by the contamination fraction $n_c/n$.
• Bootstrap ➔ resample $n$ rows with replacement, refit, $m$ times · percentiles ⟹ CI · the exact version enumerates $n^n$ ➔ Bootstrap.
• Permutation test ➔ shuffle $y$ against fixed $x$ ⟹ null of no association · $p=\frac{1+\text{count}}{m+1}$ ➔ Permutation Tests.

8️⃣ Unsupervised Learning (W11)

• Supervised vs unsupervised ➔ “is the data labelled?” · supervised models $p(Y\mid X_1,\dots ,X_p)$ — classification, regression · unsupervised finds structure, often via the joint $p(X_1,\dots ,X_q)$ — clustering, association, imputation · clustering $=$ “intrinsic classification”: discover $K$ and characterise each cluster.
• Multivariate normal $N(\mathbf{\mu },\mathbf{\Sigma })$ ➔ $\mathbf{\mu }$ moves the cloud · ${\Sigma }_{jj}$ stretches along axis $j$ · off-diagonal ${\Sigma }_{ij}=\mathrm{cov}(Y_i,Y_j)$ tilts it (positive ⟹ up-right) · ⛔ ${\Sigma }_{jj}$ is a variance, not an SD · structures: spherical ($1$ parameter) · diagonal ($q$) · full/elliptical ($q(q+1)/2$).
• k-means ➔ assign to the nearest centroid ➔ re-centre ➔ stop when nothing moves · minimises $D={\sum }_i{\min }_k{d}_k(i{)}^2$ · local minima ⟹ k-means++ seeding.
• Mixtures ➔ $p({\mathbf{y}}_i)={\sum }_k{\alpha }_{k}p({\mathbf{y}}_i\mid {\mathbf{\theta }}_k)$ · $r_{i,k}=\frac{{\alpha }_k{p}_k({\mathbf{y}}_i)}{{\sum }_l{\alpha }_l{p}_l({\mathbf{y}}_i)}$ · EM $=$ soft k-means · $K$ by IC.
• Imputation ➔ univariate vs multivariate-normal vs kNN on co-observed columns ➔ Missing Data Imputation and Recommender Systems.