Wiki
Wiki
Courses and public working groups
Courses and public working groups
Private Working Groups
Private Working Groups
- Theatre
- Admin
- Research
- Teaching
Sidebar
world:pca
$$ \newcommand{\arginf}{\mathrm{arginf}} \newcommand{\argmin}{\mathrm{argmin}} \newcommand{\argmax}{\mathrm{argmax}} \newcommand{\asconv}[1]{\stackrel{#1-a.s.}{\rightarrow}} \newcommand{\Aset}{\mathsf{A}} \newcommand{\b}[1]{{\mathbf{#1}}} \newcommand{\ball}[1]{\mathsf{B}(#1)} \newcommand{\bbQ}{{\mathbb Q}} \newcommand{\bproof}{\textbf{Proof :}\quad} \newcommand{\bmuf}[2]{b_{#1,#2}} \newcommand{\card}{\mathrm{card}} \newcommand{\chunk}[3]{{#1}_{#2:#3}} \newcommand{\condtrans}[3]{p_{#1}(#2|#3)} \newcommand{\convprob}[1]{\stackrel{#1-\text{prob}}{\rightarrow}} \newcommand{\Cov}{\mathbb{C}\mathrm{ov}} \newcommand{\cro}[1]{\langle #1 \rangle} \newcommand{\CPE}[2]{\PE\lr{#1| #2}} \renewcommand{\det}{\mathrm{det}} \newcommand{\dimlabel}{\mathsf{m}} \newcommand{\dimU}{\mathsf{q}} \newcommand{\dimX}{\mathsf{d}} \newcommand{\dimY}{\mathsf{p}} \newcommand{\dlim}{\Rightarrow} \newcommand{\e}[1]{{\left\lfloor #1 \right\rfloor}} \newcommand{\eproof}{\quad \Box} \newcommand{\eremark}{</WRAP>} \newcommand{\eqdef}{:=} \newcommand{\eqlaw}{\stackrel{\mathcal{L}}{=}} \newcommand{\eqsp}{\;} \newcommand{\Eset}{ {\mathsf E}} \newcommand{\esssup}{\mathrm{essup}} \newcommand{\fr}[1]{{\left\langle #1 \right\rangle}} \newcommand{\falph}{f} \renewcommand{\geq}{\geqslant} \newcommand{\hchi}{\hat \chi} \newcommand{\Hset}{\mathsf{H}} \newcommand{\Id}{\mathrm{Id}} \newcommand{\img}{\text{Im}} \newcommand{\indi}[1]{\mathbf{1}_{#1}} \newcommand{\indiacc}[1]{\mathbf{1}_{\{#1\}}} \newcommand{\indin}[1]{\mathbf{1}\{#1\}} \newcommand{\itemm}{\quad \quad \blacktriangleright \;} \newcommand{\jointtrans}[3]{p_{#1}(#2,#3)} \newcommand{\ker}{\text{Ker}} \newcommand{\klbck}[2]{\mathrm{K}\lr{#1||#2}} \newcommand{\law}{\mathcal{L}} \newcommand{\labelinit}{\pi} \newcommand{\labelkernel}{Q} \renewcommand{\leq}{\leqslant} \newcommand{\lone}{\mathsf{L}_1} \newcommand{\lrav}[1]{\left|#1 \right|} \newcommand{\lr}[1]{\left(#1 \right)} \newcommand{\lrb}[1]{\left[#1 \right]} \newcommand{\lrc}[1]{\left\{#1 \right\}} \newcommand{\lrcb}[1]{\left\{#1 \right\}} \newcommand{\ltwo}[1]{\PE^{1/2}\lrb{\lrcb{#1}^2}} \newcommand{\Ltwo}{\mathrm{L}^2} \newcommand{\mc}[1]{\mathcal{#1}} \newcommand{\mcbb}{\mathcal B} \newcommand{\mcf}{\mathcal{F}} \newcommand{\meas}[1]{\mathrm{M}_{#1}} \newcommand{\norm}[1]{\left\|#1\right\|} \newcommand{\normmat}[1]{{\left\vert\kern-0.25ex\left\vert\kern-0.25ex\left\vert #1 \right\vert\kern-0.25ex\right\vert\kern-0.25ex\right\vert}} \newcommand{\nset}{\mathbb N} \newcommand{\N}{\mathcal{N}} \newcommand{\one}{\mathsf{1}} \newcommand{\PE}{\mathbb E} \newcommand{\pminfty}{_{-\infty}^\infty} \newcommand{\PP}{\mathbb P} \newcommand{\projorth}[1]{\mathsf{P}^\perp_{#1}} \newcommand{\Psif}{\Psi_f} \newcommand{\pscal}[2]{\langle #1,#2\rangle} \newcommand{\pscal}[2]{\langle #1,#2\rangle} \newcommand{\psconv}{\stackrel{\PP-a.s.}{\rightarrow}} \newcommand{\qset}{\mathbb Q} \newcommand{\revcondtrans}[3]{q_{#1}(#2|#3)} \newcommand{\rmd}{\mathrm d} \newcommand{\rme}{\mathrm e} \newcommand{\rmi}{\mathrm i} \newcommand{\Rset}{\mathbb{R}} \newcommand{\rset}{\mathbb{R}} \newcommand{\rti}{\sigma} \newcommand{\section}[1]{==== #1 ====} \newcommand{\seq}[2]{\lrc{#1\eqsp: \eqsp #2}} \newcommand{\set}[2]{\lrc{#1\eqsp: \eqsp #2}} \newcommand{\sg}{\mathrm{sgn}} \newcommand{\supnorm}[1]{\left\|#1\right\|_{\infty}} \newcommand{\thv}{{\theta_\star}} \newcommand{\tmu}{ {\tilde{\mu}}} \newcommand{\Tset}{ {\mathsf{T}}} \newcommand{\Tsigma}{ {\mathcal{T}}} \newcommand{\ttheta}{{\tilde \theta}} \newcommand{\tv}[1]{\left\|#1\right\|_{\mathrm{TV}}} \newcommand{\unif}{\mathrm{Unif}} \newcommand{\weaklim}[1]{\stackrel{\mathcal{L}_{#1}}{\rightsquigarrow}} \newcommand{\Xset}{{\mathsf X}} \newcommand{\Xsigma}{\mathcal X} \newcommand{\Yset}{{\mathsf Y}} \newcommand{\Ysigma}{\mathcal Y} \newcommand{\Var}{\mathbb{V}\mathrm{ar}} \newcommand{\zset}{\mathbb{Z}} \newcommand{\Zset}{\mathsf{Z}} $$
Statement
Let $(X_i)_{1\leq i \leq n}$ be $n$ observations in $\rset^d$. Define $\Hset_p$ the set of all the subspaces of $\rset^d$ of dimension $p$ and write $\projorth{H}X$ the orthogonal projection of a vector $X \in \rset^d$ on a subspace $H$.
In Principal Component Analysis (PCA), we consider the optimisation problem: \begin{align*} V_p&= \argmin_{H \in \Hset_p} \sum_{i=1}^n \norm{X_i - \projorth{H}X_i}^2 \\ &=\argmin_{H \in \Hset_p} \sum_{i=1}^n \norm{X_i}^2 - \norm{\projorth{H}X_i}^2\\ &=\argmax_{H \in \Hset_p} \sum_{i=1}^n \norm{\projorth{H}X_i}^2 \end{align*}
Define $S_n= \sum_{i=1}^n X_i X_i^T$ and since $S_n$ is a symmetric and positive matrix with real entries, it is diagonizable in an orthonormal basis $(w_j)_{1\leq j \leq d}$ with eigenvalues $(\lambda_j)_{1\leq j \leq d}$ ranked in a decreasing order, that is, $\lambda_1 \geq \ldots \geq \lambda_n \geq 0$. Defining $D={\mathrm {Diag}} ((\lambda_j)_{1\leq j \leq d})$ and $U=[w_1,\ldots,w_d]$ we have $S_n = U D U^T=\sum_{j=1}^d \lambda_j w_j w_j^T$.
For any $p \in [1:d]$, we have
\begin{equation} \label{eq:vp}
V_p=\mathrm{Span}(w_1,\ldots,w_p)
\end{equation}
Proof
The proof is by induction. We start with $p=1$. For any unitary vector $w\in \rset^d$, \begin{align} \sum_{i=1}^n \norm{\projorth{\rset w}X_i}^2&=\sum_{i=1}^n (X_i^T w)^2=\sum_{i=1}^n w^T X_i X_i^T w=w^T (\sum_{i=1}^n X_i X_i^T) w \nonumber\\ &=w^T S_n w=w^T \lr{\sum_{j=1}^d \lambda_j w_j w_j^T} w=\sum_{j=1}^d \lambda_j (w^T w_j)^2 \label{eq:dim1} \end{align} Therefore, we have \begin{align*} \sum_{i=1}^n \norm{\projorth{\rset w}X_i}^2 &\leq \lambda_1 \sum_{j=1}^d (w^T w_j)^2=\lambda_1 \norm{w}^2=\lambda_1 \end{align*} Note that from \eqref{eq:dim1}, we have for any eigenvector $w_k$ of $S_n$, \begin{equation} \label{eq:eigenvector} \sum_{i=1}^n \norm{\projorth{\rset w_k}X_i}^2=w_k^T S_n w_k=\lambda_k. \end{equation} In particular, $\sum_{i=1}^n \norm{\projorth{\rset w_1}X_i}^2=\lambda_1$. Therefore \eqref{eq:vp} holds true for $p=1$.
Assume now that \eqref{eq:vp} hold true for some $p \in [1:d-1]$ and let $H \in \Hset_{p+1}$. Then, denote by $G=\mathrm{Span}(w_1,\ldots,w_p)^\perp$. Since $\mathrm{dim} (G)=d-p$ and $\mathrm{dim}(H)=p+1$, we must have $G \cap H \notin \{0\}$ (Otherwise the subspace $G+H$ would be of dimension $d-p+p+1=d+1$ which is not possible). Let $w_0$ a unitary vector of $G \cap H$. Then, we have the decomposition $H=\rset w_0 \stackrel{\perp}{+} H_0$ where $H_0$ is of dimension $p$. Then, applying \eqref{eq:dim1} \begin{align*} \sum_{i=1}^n \norm{\projorth{H}X_i}^2&= \sum_{i=1}^n \norm{\projorth{\rset w_0}X_i}^2+\norm{\projorth{H_0}X_i}^2\\ &=\sum_{j=1}^d \lambda_j (w_0^T w_j)^2 +\sum_{i=1}^n \norm{\projorth{H_0}X_i}^2\\ &=\sum_{j=p+1}^d \lambda_j (w_0^T w_j)^2 +\sum_{i=1}^n \norm{\projorth{H_0}X_i}^2\\ \end{align*} where we used that $w_0 \in G=\mathrm{Span}(w_1,\ldots,w_p)^\perp$. Applying the induction assumption and then \eqref{eq:eigenvector}, \begin{align*} \sum_{i=1}^n \norm{\projorth{H}X_i}^2& \leq \lambda_{p+1} \sum_{j=p+1}^d (w_0^T w_j)^2 +\sum_{i=1}^n \norm{\projorth{\mathrm{Span}(w_1,\ldots,w_p)}X_i}^2\\ & \leq \lambda_{p+1} \norm{w_0}^2+\sum_{i=1}^n \norm{\projorth{\mathrm{Span}(w_1,\ldots,w_p)}X_i}^2\\ & = \lambda_{p+1} +\sum_{i=1}^n \norm{\projorth{\mathrm{Span}(w_1,\ldots,w_p)}X_i}^2\\ & = \sum_{i=1}^n \norm{\projorth{\rset w_{p+1}}X_i}^2 +\sum_{i=1}^n \norm{\projorth{\mathrm{Span}(w_1,\ldots,w_p)}X_i}^2\\ & = \sum_{i=1}^n \lr{\norm{\projorth{\rset w_{p+1}}X_i}^2 +\norm{\projorth{\mathrm{Span}(w_1,\ldots,w_p)}X_i}^2}\\ & = \sum_{i=1}^n \norm{\projorth{\mathrm{Span}(w_1,\ldots,w_{p+1})}X_i}^2\\ \end{align*} which concludes the proof by an induction argument.
world/pca.txt · Last modified: 2022/11/13 18:38 by rdouc
