Skip to content

Multivariate Distributions

Published July 26, 2022

Multivariate Distributions

Random Vectors

Definition: A vector $X$ is called a random vector if it has an associated pdf $f(.)$
\begin{eqnarray}
X =\left[
\begin{array}{c}
x_1\\
\vdots\\
x_n
\end{array}
\right] \sim f(x_1,x_2,\ldots,x_n)
\end{eqnarray}

where each $x_i$ is a random variable. We can say that random vectors are multidimensional extensions of random variables.

Expectation vector, Covariance Matrix and Correlation Matrix

Each random variable has a mean and a variance. In the case of random vectors, these concepts are replaced by expectation vector and covariance matrix.

The expectation of a random vector is defined as
\begin{eqnarray}
\mathrm{E}(X) =\left[
\begin{array}{c}
\mathrm{E}(x_1)\\
\vdots\\
\mathrm{E}(x_n)
\end{array}
\right] = \mu=\left[
\begin{array}{c}
\mu_1\\
\vdots\\
\mu_n
\end{array}
\right]
\end{eqnarray}
where
\begin{eqnarray}
\mathrm{E}(x_i)=\int x_i f(x_1,\ldots, x_n)dx_1\ldots dx_n
\end{eqnarray}
The covariance matrix of the random vector is defined as
\begin{eqnarray}
\mathrm{Cov}(X) =\Sigma=\left[
\begin{array}{cccc}
V(x_1) & \mathrm{Cov}(x_1,x_2) & \ldots & \mathrm{Cov}(x_1,x_n)\\
\mathrm{Cov}(x_1,x_2) & V(x_2) & \ldots & \mathrm{Cov}(x_2,x_n)\\
\vdots & \vdots & \ddots & \vdots\\
\mathrm{Cov}(x_1,x_n) & \mathrm{Cov}(x_2,x_n) & \ldots & V(x_n)
\end{array}
\right]
\end{eqnarray}
where $V(x_i)=\sigma_i^2$ denotes the variance of the random variable $x_i$:
\begin{eqnarray}
V(x_i)&=&\mathrm{E}(x_i-\mathrm{E}(x_i))^2\\
&=&\mathrm{E}(x_i-\mu_i)^2 \\
&=&\mathrm{E}(x_i^2)-\mu_i^2\\
&=& \sigma_i^2
\end{eqnarray}
Recall that the square root of variance (ie, standard deviation) measures the spread of a random variable around its mean.

$\mathrm{Cov}(x_i,x_j)$ denotes the “relatedness” between the random variables $x_i$ and $x_j$ and is defined as
\begin{eqnarray}
\mathrm{Cov}(x_i,x_j)&=&\mathrm{E}[(x_i-\mathrm{E}(x_i))(x_i-\mathrm{E}(x_j))]\\
&=&\mathrm{E}[(x_i-\mu_i)(x_j-\mu_j)] \\
&=&\mathrm{E}[x_i x_j] -\mu_i\mu_j\\
&=& \sigma_{ij}.
\end{eqnarray}

Affine Transformations of Random Vectors

Theorem: Let $X$ be a random vector with $E(X)=\mu$ and $V(X)=\Sigma$. Define the random variable $Y$ as $Y=a+bX$. Then
\begin{eqnarray}
\mathrm{E}(Y)=a+b\mu\\
\mathrm{Cov}(Y)=\Sigma b \Sigma^T
\end{eqnarray}

Correlation Matrix

Orthogonality

Independence

Multivariate Normal distribution

If the pdf associated with a random vector is normal, then it is called a normal random vector, and denoted by $N(\mu,\Sigma)$. An example
is the 2D normal pdf
\begin{eqnarray}
x= \left[
\begin{array}{c}
x_1\\
\vdots\\
x_n
\end{array}
\right] \sim \frac{1}{\sqrt{(2 \pi)^n |\Sigma|}} e^{-\frac{1}{2}(x-\mu)^T \Sigma^{-1}(x-\mu)}
\end{eqnarray}
where $|.|$ denotes determinant, $\mu$ is a $n \times 1$ vector,
\begin{eqnarray}
\mu = \left[
\begin{array}{c}
\mu_1\\
\vdots\\
\mu_n
\end{array}
\right]
\end{eqnarray}
and $\Sigma$ is an $n \times n$ symmetric matrix
\begin{eqnarray}
\Sigma = \left[
\begin{array}{cccc}
\sigma_{1}^2 & \sigma_{12} & \ldots & \sigma_{1n}\\
\sigma_{12} & \sigma_{2}^2 & \ldots & \sigma_{2n}\\
\vdots & \ddots & \vdots & \vdots \\
\sigma_{1n} & \sigma_{2n} & \ldots & \sigma_{n}^2\\
\end{array}
\right]
\end{eqnarray}

Properties of Multivariate normal Distribution

The following facts is provided without proof:

  1. The function (xx) integrates to 1, hence it is a pdf.
  2. The vector $\mu$ is the mean
    \begin{eqnarray}
    \mathrm{E}(x)=\mu\\
    \mathrm{E}(x_i)=\mu_i
    \end{eqnarray}.
  3. The matrix $\Sigma$ is the Covariance Matrix.
    \begin{eqnarray}
    \mathrm{E}[(x-\mu)(x-\mu)’]=\Sigma\\
    \mathrm{E}[(x_i-\mu_i)(x_j-\mu_j)]=\sigma_{ij}
    \end{eqnarray}.

Marginal and Conditional Distributions

Let us do the following partitions:

  • Partition the $n$-dimensional random vector x into subvectors $x_1$ and $x_2$
    \begin{eqnarray}
    x = \left[
    \begin{array}{c}
    x_1\\
    x_2
    \end{array}
    \right]
    \end{eqnarray}
    where $x_1$ is $p$ dimensional, $x_2$ is $q$ dimensional, and $p+q=n$.
  • Partition the $n$-dimensional expectation vector $\mu$ subvectors $\mu_1$ and $\mu_2$ , .
    \begin{eqnarray}
    \mu = \left[
    \begin{array}{cc}
    \mu_1\\
    \mu_2
    \end{array}
    \right]
    \end{eqnarray}
    where $\mu_1$ is $p$ dimensional and $\mu_2$ is $q$ dimensional.
  • Partition the $n \times n$ covariance matrix $\Sigma$ into four submatrices:
    \begin{eqnarray}
    \Sigma = \left[
    \begin{array}{cc}
    \Sigma_{11} & \Sigma_{12}\\
    \Sigma_{21} & \Sigma_{22}
    \end{array}
    \right]
    \end{eqnarray}where $\Sigma_{11}$ is $p \times p$ dimensional, $\Sigma_{12}$ is $p \times q$ dimensional,
    $\Sigma_{21}$ is $q \times p$ dimensional and $\Sigma_{22}$ is $q \times q$ dimensional.

Then the marginal distribution is

\begin{eqnarray}
x_1 \sim \mathrm{N}(\mu_1, \Sigma_{11})
\end{eqnarray}

and the conditional distribution is

\begin{eqnarray}
x_2|x_1 \sim \mathrm{N}(a+B’x_1, \Sigma_{22}-B’\Sigma_{11}B)
\end{eqnarray}

where $B=(\Sigma_{11}^{-1}\Sigma_{12})$ and $a=\mu_2-B’\mu_1$

Categories:probability and statistics