Welcome to Randal Douc's wiki

You've loaded an old revision of the document! If you save it, you will create a new version with this data.

=== Bayes Optimal Classifier ===

Let $(X,Y)$ be random vector on $ (\Omega,\mcf,\PP) $, taking values on $\rset^k \times [1:p]$. We are interested in solving the optimization problem 
$$
\inf_{\phi \in \sf{F}} \PP(Y\neq \phi(X))
$$
where $\sf{F}$ is the set of measurable functions from $\rset^k$ to $[1:p]$ where we equip $\rset^k$ with the $\sigma$-field $\mcbb(\rset^k)$ and $[1:p]$ with the $\sigma$-field $\mc{P}([1:p])$.

Proposition

$$
\inf_{\phi \in \sf{F}} \PP(Y\neq \phi(X))= \PE\lrb{\min_{i \in [1:p]} \PP(Y \neq i|X)}=\PP(Y \neq \phi^\star(X))
$$
where $\phi^\star(X)=\argmax_{i \in [1:p]} \PP(Y=i|X)$.

Proof

For any classifier $\phi$, decomposing into the disjoint events $\{\phi(X)=i\}$ and using the tower property, 
\begin{align*}
    \PP(Y\neq \phi(X)) &= \sum_{i=1}^p \PP(\phi(X)=i, Y \neq i)=\sum_{i=1}^p \PE\lrb{\indiacc{\phi(X)=i} \PP(Y \neq i|X)} \\ 
    & =\PE\lrb{\sum_{i=1}^p \indiacc{\phi(X)=i} \PP(Y \neq i|X)} \geq \PE\lrb{\min_{i\in [1:p]} \PP(Y \neq i|X)}
\end{align*}
Moreover the inequality above becomes an equality for $\phi=\phi^\star$ which satisfies the property: 
$$
\phi^\star(X)= \argmin_{i \in [1:p]} \PP(Y \neq i|X)=\argmax_{i \in [1:p]} \PP(Y=i|X)
$$
which concludes the proof.

Please fill all the letters into the box to prove you're human. L A O Q E Please keep this field empty:

Edit summary

Welcome to Randal Douc's wiki

User Tools

Site Tools

Page Tools