Welcome to Randal Douc's wiki

You've loaded an old revision of the document! If you save it, you will create a new version with this data.

====== Kullback-Leibler divergence for normal distributions ======

<WRAP center round tip 90%>
Let $\mu_0,\mu_1 \in \rset^p$ and $ \Sigma_0,\Sigma_1  \in \rset^{p \times p}$ where $\Sigma_0,\Sigma_1$ are symmetric definite positive. Then,  
$$
\klbck{\N(\mu_0,\Sigma_0)}{\N(\mu_1,\Sigma_1)}=-\frac{p}{2}  + \frac{(\mu_0-\mu_1)^T \Sigma_1^{-1} (\mu_0-\mu_1) }{2} + \frac{Tr\lr{\Sigma_1^{-1} \Sigma_0}}{2} - \frac{1}{2} \log \frac{\mathrm{det} \Sigma_0}{\mathrm{det} \Sigma_1}
$$

</WRAP>

===== Proof =====

Assume that $X_0\sim \N(\mu_0,\Sigma_0)$ then $X_0= \mu_0 + \Sigma_0^{1/2} U_0$ where $U_0\sim \N(0,I_p)$
\begin{align*}
    \klbck{\N(\mu_0,\Sigma_0)}{\N(\mu_1,\Sigma_1)}&= \PE\lrb{\log \frac{\rme^{-(X_0-\mu_0)^T \Sigma_0^{-1} (X_0-\mu_0) / 2}}{\rme^{-(X_0-\mu_1)^T \Sigma_1^{-1} (X_0-\mu_1) / 2}}} - \frac{1}{2} \log \frac{\mathrm{det} \Sigma_0}{\mathrm{det} \Sigma_1} \\ 
    & = \PE\lrb{-\frac{U_0^T U_0}{2}}+\frac{1}{2} \PE\lrb{(\mu_0-\mu_1 + \Sigma_0^{1/2}U_0)^T \Sigma_1^{-1} (\mu_0-\mu_1 + \Sigma_0^{1/2}U_0)} - \frac{1}{2} \log \frac{\mathrm{det} \Sigma_0}{\mathrm{det} \Sigma_1} \\ 
    &= -\frac{p}{2}  + \frac{(\mu_0-\mu_1)^T \Sigma_1^{-1} (\mu_0-\mu_1) }{2} + \frac 1 2 \PE\lrb{U_0^T \Sigma_0^{1/2} \Sigma_1^{-1} \Sigma_0^{1/2} U_0} - \frac{1}{2} \log \frac{\mathrm{det} \Sigma_0}{\mathrm{det} \Sigma_1}\\ 
    &= -\frac{p}{2}  + \frac{(\mu_0-\mu_1)^T \Sigma_1^{-1} (\mu_0-\mu_1) }{2} + \frac 1 2 \PE\lrb{U_0^T \Sigma_0^{1/2} \Sigma_1^{-1} \Sigma_0^{1/2} U_0} - \frac{1}{2} \log \frac{\mathrm{det} \Sigma_0}{\mathrm{det} \Sigma_1}\\ 
    &= -\frac{p}{2}  + \frac{(\mu_0-\mu_1)^T \Sigma_1^{-1} (\mu_0-\mu_1) }{2} + \frac{Tr\lr{\Sigma_1^{-1} \Sigma_0}}{2} - \frac{1}{2} \log \frac{\mathrm{det} \Sigma_0}{\mathrm{det} \Sigma_1}
\end{align*}

Please fill all the letters into the box to prove you're human. E I O E᠎ O Please keep this field empty:

Edit summary

Welcome to Randal Douc's wiki

User Tools

Site Tools

Page Tools