Let be a matrix with real-valued entries. We define and . We can note that and are subspaces of .

. Otherwise stated, $x$ is the orthogonal projection of $y$ on $\b{X}$ if and only if we have the two properties $x \in \img(\bf{X})$ and $(y-x) \in \ker(\b{X}^T)$. The fact that is the orthogonal complement of stems from the following remark: is equivalent to the fact that where are the column vectors of and this in turn is equivalent to .

Denote by the matrix of the orthogonal projection on . By abuse of notation, we also write . We have and we can also note that on .

An orthogonal projection is uniquely determined by the subspace on which it projects. This implies in particular the following property. Assume that is a matrix and is a matrix where we can possibly have . As soon as , we have

By the Pythagore identity, for all , we have () showing that with equality only if .

is a pseudo inverse of (and we note ) iff or, equivalently, for all , . We admit that a pseudo inverse always exists.

Let be a vector of size and a matrix. Since is in , it can be written as for some .

A side effect of this theorem is that does not depend on the choice of the pseudo inverse .

We can now give the general solutions of the normal equations.

We now consider where is a zero mean vector with covariance matrix . We say that is estimable if there exists such that is an unbiased estimator of , which is equivalent to for all and therefore to $\lambda^T=a^T\b{X}$.

In the curse of the proof, we have seen that if is estimable, then choosing such that , and a side effect is that does not depend on the chosen pseudo-inverse whenever .

Video of a short course on the Gauss Markov Theorem