Limit Theorems
Weak law of large numbers: casinos work for small mean but large variance–a lot of nerve is required..
Central Limit Theorem
In this section, we will see why Gaussian distribution is so important..
Let $X_1,X_2,\ldots,X_n$ be $n$ iid random variables, with finite expectation $E$ and variance $\sigma^2$. Beyond mean and variance, we assume to know nothing about their distribution. Define a new random variable $R$ as the sum of $X_1,X_2,\ldots,X_n$, ie,
\begin{eqnarray}
R= X_1+X_2+\ldots+X_n
\end{eqnarray}Can we say anything about the pdf of $R$, $f_R(r)$, without knowing anything about the pdf
of $X_i$’s beyond the mean and variance? Central limit theorem claims yes..
Central Limit Theorem:
\begin{eqnarray}
\lim_{n \rightarrow \infty} P_R(r)=\Phi\left( \frac{r-nE}{\sqrt{n}\sigma}\right)
\end{eqnarray}
where $P_R(.)$ is the cdf of the random variable $R$, and $\Phi(.)$ is the cdf for the standard normal distribution.
Proof: Before delving into the proof, let us demonstrate a preliminary result: If $R$ is a random variable with pdf $f_R(r)$ and mgf $f_R^T(s)$,
\begin{eqnarray}
f_R^T(s)=\int_0^{\infty}e^{-rs}f_R(r)dr
\end{eqnarray}
then the random variable $Y=aR+b$, where $a,b \in \mathbb{R}$, has the pdf
\begin{eqnarray}
f_Y(y)=\frac{d}{dy} \int_{-\infty}^{\frac{y-b}{a}}f_R(r)dr = \frac{1}{|a|}f_R\left(\frac{y-b}{a}\right) = \frac{1}{|a|} f_R(r)
\end{eqnarray}
mgf $f_Y^T(s)$
\begin{eqnarray}
f_Y^T(s)=\int_0^{\infty}e^{-ys}f_Y(y)dy = \int_0^{\infty} e^{-(ar+b)s}\frac{1}{|a|}f_R(r)|a|dr = e^{-sb}f_R^T(as)
\end{eqnarray}
Now, if
\begin{eqnarray}
R= X_1+X_2+\ldots+X_n
\end{eqnarray}
as defined above, then
\begin{eqnarray}
f_R^T(s) = \left[ f_X^T(s) \right]^n
\end{eqnarray}
Let us define the new random variable $Y$
\begin{eqnarray}
Y = \frac{X_1+\ldots+X_n-nE}{\sqrt{n}\sigma} = \frac{R-nE}{\sqrt{n}\sigma}
\end{eqnarray}
This is a transformation of the sort $Y=aR+b$ with
\begin{eqnarray}
a&=&\frac{1}{\sqrt{n}\sigma}\\
b&=&\frac{-nE}{\sqrt{n}\sigma} = \frac{-\sqrt{n}E}{\sigma}
\end{eqnarray}
which results in
\begin{eqnarray}
f_Y^T(s)&=& e^{-\frac{-\sqrt{n}E}{\sigma}} f_R^T(\frac{1}{\sqrt{n}\sigma}s) \\
&=& e^{\frac{\sqrt{n}E}{\sigma}} \left[ f_X^T(\frac{1}{\sqrt{n}\sigma}s) \right]^n \\
&=& \left[ e^{\frac{E}{\sqrt{n}\sigma}} f_X^T(\frac{1}{\sqrt{n}\sigma}s) \right]^n
\end{eqnarray}
Now, let us expand the factors of the above expression into taylor series around zero:
\begin{eqnarray}
e^{\frac{E}{\sqrt{n}\sigma}} &=& 1 + \frac{E}{\sqrt{n}\sigma} + \frac{1}{2}\left[ \frac{E}{\sqrt{n}\sigma} \right]^2 + \mathrm{h.o.t.} \nonumber\\
\nonumber
f_X^T(\frac{1}{\sqrt{n}\sigma}s) &=& f_X^T(0) + \frac{d}{ds}\left[ f_X^T(\frac{1}{\sqrt{n}\sigma}s) \right]_{s=0}s +
\frac{1}{2}\left(\frac{d^2}{ds^2}\left[ f_X^T(\frac{1}{\sqrt{n}\sigma}s) \right]\right)_{s=0}s^2+ \mathrm{h.o.t.} \nonumber
\end{eqnarray}\\
Define new parameter $u=\frac{1}{\sqrt{n}\sigma}s$. \\
\begin{eqnarray}
f_X^T(\frac{1}{\sqrt{n}\sigma}s) &=& f_X^T(0) + \left( \frac{df_X^T(u)}{du} \frac{du}{ds} \right)_{s=0} s
+\left[ \frac{1}{2}\frac{d}{du} \left(\frac{d f_X^T(u)}{du}\frac{du}{ds} \right)\frac{du}{ds} \right]_{s=0} s^2 + \mathrm{h.o.t.} \nonumber
\end{eqnarray} \\
Note that $\frac{du}{ds}=\frac{1}{\sqrt{n}\sigma}$ and when $s=0$, alsı $u=0$. Hence \\
\begin{eqnarray}
f_X^T(\frac{1}{\sqrt{n}\sigma}s) &=& f_X^T(0) +\frac{1}{\sqrt{n}\sigma} \left( \frac{df_X^T(u)}{du}\right)_{u=0} s
+ \frac{1}{2n\sigma^2} \left[\frac{d^2 f_X^T(u)}{du^2}\right]_{u=0} s^2 + \mathrm{h.o.t.} \nonumber
\end{eqnarray}\\
Now recall from the theory of moment generating functions
\begin{eqnarray}
f_X^T(0) &=& 1 \\
\left( \frac{df_X^T(u)}{du}\right)_{u=0} &=& -E[u] = -E\\
\left[\frac{d^2 f_X^T(u)}{du^2}\right]_{u=0} &=& E[u^2] = \sigma^2+E^2
\end{eqnarray}\\
Substituting, we get
\begin{eqnarray}
f_X^T(\frac{1}{\sqrt{n}\sigma}s) &=& 1-\frac{E}{\sqrt{n}\sigma}s + \frac{\sigma^2+E^2}{2n\sigma^2} s^2 + \mathrm{h.o.t.} \nonumber
\end{eqnarray}
Then
\begin{eqnarray}
e^{\frac{E}{\sqrt{n}\sigma}}f_X^T(\frac{1}{\sqrt{n}\sigma}s) = \left[ 1 + \frac{E}{\sqrt{n}\sigma}s + \frac{1}{2} \frac{E^2}{n\sigma^2}s^2 + \mathrm{h.o.t.}\right]
\left[1-\frac{E}{\sqrt{n}\sigma}s + \frac{\sigma^2+E^2}{2n\sigma^2} s^2 + \mathrm{h.o.t.} \right] \nonumber
\end{eqnarray}
Keeping only the terms of $O(s^2)$, we arrive at
\begin{eqnarray}
e^{\frac{E}{\sqrt{n}\sigma}}f_X^T(\frac{1}{\sqrt{n}\sigma}s) &=& 1+ \frac{1}{2} \frac{E^2}{n\sigma^2}s^2 +\frac{\sigma^2+E^2}{2n\sigma^2} s^2
-\frac{E^2}{n\sigma^2}s^2+\mathrm{h.o.t.}\nonumber\\
&=&1+\frac{s^2}{2n}+\mathrm{h.o.t.}\nonumber
\end{eqnarray}
Hence
\begin{eqnarray}
f_Y^T(s)&=& \left[ e^{\frac{E}{\sqrt{n}\sigma}} f_X^T(\frac{1}{\sqrt{n}\sigma}s) \right]^n \\
&\approx&\left[ 1+\frac{s^2}{2n} \right]^n\\
&\approx&e^{\frac{s^2}{2}}
\end{eqnarray}
for large $n$. But this is the characteristic function of standard normal distribution, which proves our theorem.