Multivariate Normal Distribution

Context: FIT2086_MOC · W11 · the Gaussian Distribution in $q$ dimensions with correlation between attributes · $\Sigma$ entries are the covariances · the class model inside Mixture Models (EM Algorithm) · the correlation engine of multivariate-normal imputation

Quick Revision

• 🎯 Objective: $\mathbf{Y}=(Y_1,\dots ,Y_q)\sim N(\mathbf{\mu },\mathbf{\Sigma })$ ➔ ${\mu }_j$ is the mean of $Y_j$, ${\Sigma }_{i,j}=\mathrm{cov}(Y_i,Y_j)$ ➔ read a scatter’s centre, spread per axis, and tilt straight off $\mathbf{\mu }$ and $\mathbf{\Sigma }$.
• ⚡ Key Constraint: diagonal entries are variances, not standard deviations (${\Sigma }_{1,1}=2\Rightarrow \mathrm{sd}=\sqrt{2}$); off-diagonals are covariances and $\mathbf{\Sigma }$ is symmetric.

📝 Core

• Density ➔ $p(\mathbf{y}\mid \mathbf{\mu },\mathbf{\Sigma })={\left(\frac{1}{2\pi }\right)}^{q/2}\sqrt{|{\mathbf{\Sigma }}^{-1}|}\exp \left(-\frac{1}{2}(\mathbf{y}-\mathbf{\mu }{)}^{\prime }{\mathbf{\Sigma }}^{-1}(\mathbf{y}-\mathbf{\mu })\right)$.
• Reduces to the univariate normal ➔ $q=1$, $\mathbf{\Sigma }={\sigma }^2$: $\sqrt{1/{\sigma }^2}=1/\sigma$ ⟹ $\frac{1}{\sqrt{2\pi }\sigma }{e}^{-(y-\mu {)}^2/(2{\sigma }^2)}$.
• Why it matters ➔ an individual’s attributes are related (height ↔ weight); separate univariate models per attribute cannot express that, $\mathbf{\Sigma }$ can.
• Covariance structures ➔ diagonal, all variances equal (spherical, $1$ variance parameter) · diagonal, variances differ ($q$) · arbitrary (elliptical, $q(q+1)/2$) ➔ each more flexible but with more parameters to estimate.
• Quadratic form = distance ➔ $(\mathbf{y}-\mathbf{\mu }{)}^{\prime }{\mathbf{\Sigma }}^{-1}(\mathbf{y}-\mathbf{\mu })$ is the Mahalanobis-type distance a Gaussian mixture uses in place of Euclidean.

📊 Reading $\mathbf{\Sigma }$ from a scatter

$\mathbf{\mu }$	$\mathbf{\Sigma }$	Cloud shape
$(0,0)$	$\left(\begin{array}{cc}1& 0\\ 0& 1\end{array}\right)$	circle at the origin
$(2,0)$	$\left(\begin{array}{cc}1& 0\\ 0& 1\end{array}\right)$	same circle shifted right — $\mathbf{\mu }$ moves, shape fixed
$(0,0)$	$\left(\begin{array}{cc}2& 0\\ 0& 1\end{array}\right)$	stretched along $y_1$
$(0,0)$	$\left(\begin{array}{cc}1& 0\\ 0& 1.5\end{array}\right)$	stretched along $y_2$
$(0,0)$	$\left(\begin{array}{cc}1& 0.6\\ 0.6& 1\end{array}\right)$	ellipse tilted up-right (positive covariance)
$(0,0)$	$\left(\begin{array}{cc}1& -0.9\\ -0.9& 1\end{array}\right)$	thin ellipse tilted down-right (strong negative)

⚠️ Common Mistakes

• 💡 Reading ${\Sigma }_{j,j}$ as an sd ➔ it is $\mathrm{Var}(Y_j)$; R’s rnorm wants the sd, $\mathbf{\Sigma }$ holds variances.
• 💡 Zero off-diagonals read as “no structure” ➔ only the tilt disappears; unequal diagonals still stretch the cloud along an axis.

🧠 Active Recall

When does a mixture of multivariate normals beat one with independent univariate normals per attribute, and what does it cost?

Answer

• Short answer: when attributes are correlated within a class (tilted clusters); an arbitrary $\mathbf{\Sigma }$ captures the tilt but costs $q(q+1)/2$ covariance parameters per class instead of $q$.
• Why: Flexibility vs complexity ➔ the extra parameters must pay for themselves in fit under the information criterion.