Chapter 1 introduction, exercise 1-10 solutions

1. (*)www Consider the sum-of-squares error function given
   $$E(w) = \frac{1}{2}\sum_{n=1}^{N}\{y(x_n,w) - t_n\}^2 \tag{1.2}$$

   in which the function $y(x,w)$ is given by the polynomial $y(x,w)=\sum_{j=0}^{M}w_jx^j$ . Show that the coefficents $w=\{w_i\}$ that minimize this error function are given by the solution to the following set of linear equations

   $$\sum_{j=0}^{M}A_{ij}w_j = T_i \tag{1.122}$$

   where

   $$A_{ij}=\sum_{n=1}^{N}(x_n)^{i+j}, T_i = \sum_{n=1}^{N}(x_n)^it_n \tag{1.123}$$

   Solution:

   $$\begin{equation} \frac{\partial E(w)}{w_i} = \sum_{n=1}^{N}(\sum_{j=0}^{M}w_jx_{n}^j - t_n)x_{n}^i=0 \\ \sum_{n=1}^{N}\sum_{j=0}^{M}w_{j}x_{n}^jx_{n}^i = \sum_{n=1}^{N}t_nx_{n}^i \\ \sum_{j=0}^{M}A_{ij}w_j = T_i \end{equation}$$

2. (*) Write down the set of coupled linear equations, analogous to $(1.122)$ , satisfied by the coefficients $w_i$ which minimize the regularized sum-of-squares error function given by $\tilde{E}(w)= \frac{1}{2}\sum_{n=1}^{N}\{y(x_n,w) - t_n\}^2 + \frac{\lambda}{2} \parallel w \parallel^2$

   Solution:

   $$\begin{equation} \frac{\partial E(w)}{w_i} = \sum_{n=1}^{N}(\sum_{j=0}^{M}w_jx_{n}^j - t_n)x_{n}^i + \lambda w_i=0 \\ \sum_{n=1}^{N}\sum_{j=0}^{M}w_{j}x_{n}^jx_{n}^i + \lambda w_i = \sum_{n=1}^{N}t_nx_{n}^i \\ \sum_{j=0}^{M}A_{ij}w_j +\lambda w_i= T_i \end{equation}$$

3. (**) Suppose that we have three coloured boxes $r$ (red), $b$ (blue), and $g$ (green). Box $r$ contains 3 apples, 4 oranges, and 3 limes, box $b$ contains 1 apple, 1 orange, and 0 limes, and box $g$ contains 3 apples, 3 oranges, and 4 limes. If a box is choosen at random with probabilities $p(r) = 0.2, p(b) = 0.2, p(g) = 0.6$, and a piece of fruit is removed from the box (with equal probability of selecting any of the items in the box), then what is the probability of selecting an apple? If we observe that the selected fruit is in fact an orange, what is the probability that it came from the green box?

   Solution: Bayes’ theorem

   $$\begin{aligned} p(apple) &= p(apple|r)p(r) + p(apple|b)p(b)+p(apple|g)p(g) \\ &= 0.3 \times 0.2 + 0.5 \times 0.2 + 0.3 \times 0.6 \\ &= 0.34 \\ p(o) &= p(o|r)p(r) + p(o|b)p(b)+p(o|g)p(g) \\ &= 0.4 \times 0.2 + 0.5 \times 0.2 + 0.3 \times 0.6\\ &= 0.36 \\ p(g|o) &= \frac{p(o|g)p(g)}{p(o)} \\ &=\frac{0.3 \times 0.6}{0.36} \\ &= 0.5 \end{aligned}$$
4. (*)www Consider a probability density $p_{x}(x)$ defined over a continuous variable $x$,
   and suppose that we make a nonlinear change of variable using $x = g(y)$, so that the
   density transforms according to
   $$p_{y}(y) = p_{x}(x)|\frac{dx}{dy}| = p_{x}(g(h))|g^{'}(y)| \tag{1.27}$$

   By differentiating (1.27), show that the location $\hat{y}$ of the maximum of the density in $y$ is not in general related to the location $\hat{x}$ of the maximum of the density over $x$ by the simple functional relation $\hat{x}=g(\hat{y})$ as a consequence of the Jacobian factor. This shows that the maximum of a probability density (in contrast to a simple function) is dependent on the choice of variable. Verify that, in the case of a linear transformation, the location of the maximum transforms in the same way as the variable itself.

   Solution: TODO

5. (*) Using the definition
   $$var[f] = \mathbb{E}[(f(x)- \mathbb{E}[f(x)])^2] \tag{1.38}$$

   show that $var[f(x)]$ satisfies

   $$var[f] = \mathbb{E}[f(x)^2] - \mathbb{E}[f(x)]^2 \tag{1.39}$$

   Solution:

   $$\begin{aligned} var[f] &= \mathbb{E}[(f(x)- \mathbb{E}[f(x)])^2] \\ &= \mathbb{E}[(f(x)^2 - 2f(x)\mathbb{E}[f(x)] + {\mathbb{E}[f(x)]}^2)] \\ &= \mathbb{E}[f(x)^2] - 2 \times \mathbb{E}[f(x)] \times \mathbb{E}[f(x)] + {\mathbb{E}[f(x)]}^2 \\ &= \mathbb{E}[f(x)^2]- {\mathbb{E}[f(x)]}^2 \end{aligned}$$

6. (*) Show that if two variables $x$ and $y$ are independent, then their covariance is zero.

   Solution:

   If $f=xy$ is a discrete distribution, then

   $$\mathbb{E}[xy] = \sum_{xy}p(xy)xy =\sum_{x}p(x)x\sum_{y}p(y)y =\mathbb{E}[x]\mathbb{E}[y]$$

   If $f=xy$ is a continuous distribution, then

   $$\mathbb{E}[xy] = \int\int p(xy)xy dxdy =\int p(x)x dx\int{p(y)ydy} =\mathbb{E}[x]\mathbb{E}[y]$$
   $$\begin{aligned} cov[x, y] &= \mathbb{E}_{x,y}[xy] - \mathbb{E}[x]\mathbb{E}[y] \\ &= \mathbb{E}[x]\mathbb{E}[y] - \mathbb{E}[x]\mathbb{E}[y] \\ &= 0 \end{aligned}$$
7. (**) www In this exercise, we prove the normalization condition (1.48) for
   the univariate Gaussian. To do this consider, the integral
   $$I = \int_{-\infty}^{\infty} exp(-\frac{1}{2\sigma^2}x^2) dx \tag{1.124}$$

   which we can evaluate by first writing its square in the form

   $$I^2 = \int_{-\infty}^{\infty}\int_{-\infty}^{\infty}exp(-\frac{1}{2\sigma^2}x^2-\frac{1}{2\sigma^2}y^2)dxdy \tag{1.125}$$

   Now make the transformation from Cartesian coordinates $(x,y)$ to the polar coordinates $(r, \theta)$ and then subsitute $u=r^2$. Show that, by performing the integrals over $\theta$ and $u$, and then taking the square root of both sides, we obtain

   $$I=(2\pi\sigma^2)^\frac{1}{2} \tag{1.126}$$

   Finally, use this result to show that the Gaussian distribution $\mathcal{N}(x|\mu, \sigma^2)$ is normalized.

   Solution:

   with $x=r \cos \theta, y=r \sin \theta$ and polar coordinates transformation $\int\int_{R}f(x,y)dA = \int\int f(r \cos \theta, r \sin \theta)rdrd\theta$,
   Why $dA=rdrd\theta$ ? check this link

   $$\begin{aligned} I^2 &= \int_{-\infty}^{\infty}\int_{-\infty}^{\infty}exp(-\frac{1}{2\sigma^2}x^2-\frac{1}{2\sigma^2}y^2)dxdy \\ &= \int_{0}^{2\pi}\int_{0}^{\infty}exp(-\frac{r^2}{2\sigma^2})rdrd\theta \\ &= \theta\rvert_{0}^{2\pi}\times (-\sigma^2)exp(-\frac{r^2}{2\sigma^2})\rvert_{0}^{\infty} \\ &= 2\pi(-\sigma^2(0-1)) \\ &= 2\pi\sigma^2 \end{aligned}$$

   so, $I=(2\pi\sigma^2)^\frac{1}{2}$, then using the transformation $y=x-\mu$

   $$\begin{aligned} \int_{-\infty}^{\infty}\mathcal{N}(x|\mu, \sigma^2)dx &= \int_{-\infty}^{\infty}\frac{1}{\sqrt{2\pi\sigma^2}}exp\{-\frac{1}{2\sigma^2}(x-\mu)^2\} dx \\ &= \int_{-\infty}^{\infty}\frac{1}{\sqrt{2\pi\sigma^2}}exp\{-\frac{1}{2\sigma^2}(y)^2\} dy \\ &= \frac{1}{\sqrt{2\pi\sigma^2}} \times I = 1 \end{aligned}$$
8. (**)www By using a change of variables, verify that the univariate Gaussian
   distribution given by
   $$\mathcal{N}(x|\mu, \sigma^2) = \frac{1}{\sqrt{2\pi\sigma^2}}exp\{-\frac{1}{2\sigma^2}(x-\mu)^2\} \tag{1.46}$$

   satisfies

   $$\mathbb{E}[x] = \int_{-\infty}^{\infty}\mathcal{N}(x|\mu, \sigma^2)xdx = \mu \tag{1.49}$$

   Next, by differentiating both sides of the normalization condition

   $$\int_{-\infty}^{\infty} \mathcal{N} (x|\mu, \sigma^2) dx = 1 \tag{1.127}$$

   with respect to $\sigma^2$, verfiy that the Gaussian satisfies

   $$\mathbb{E}[x^2] = \int_{-\infty}^{\infty}\mathcal{N}(x|\mu, \sigma^2)x^2dx = \mu^2 + \sigma^2 \tag{1.50} \\ $$
   $$var[x] = \mathbb{E}[x^2] - \mathbb{E}[x]^2 = \sigma^2 \tag{1.51}$$

   Finally, show that (1.51) holds.

   Solution:

   with $y=x-\mu$ and integral of an odd function is 0 $f$ is odd if $-f(x) = f(-x)$

   $$\begin{aligned} \mathbb{E}[x] &= \int_{-\infty}^{\infty}\mathcal{N}(x|\mu, \sigma^2)x dx \\ &= \int_{-\infty}^{\infty}\frac{1}{\sqrt{2\pi\sigma^2}}exp\{-\frac{1}{2\sigma^2}(x-\mu)^2\}x dx \\ &= \int_{-\infty}^{\infty}\frac{1}{\sqrt{2\pi\sigma^2}}exp\{-\frac{1}{2\sigma^2}(y)^2\}(y+\mu) dy \\ &= \int_{-\infty}^{\infty}\frac{1}{\sqrt{2\pi\sigma^2}}exp\{-\frac{1}{2\sigma^2}(y)^2\}(y) dy + \int_{-\infty}^{\infty}\frac{1}{\sqrt{2\pi\sigma^2}}exp\{-\frac{1}{2\sigma^2}(y)^2\}(\mu) dy \\ &= 0 + \mu\int_{-\infty}^{\infty}\frac{1}{\sqrt{2\pi\sigma^2}}exp\{-\frac{1}{2\sigma^2}(y)^2\}dy \\ &= \mu \end{aligned}$$

   by differentiating both sides of the normalization condition (1.127) with respect to $\sigma^2$

   $$\begin{aligned} \int_{-\infty}^{\infty} \mathcal{N} (x|\mu, \sigma^2) dx = \int_{-\infty}^{\infty} \frac{1}{\sqrt{2\pi\sigma^2}}exp\{-\frac{1}{2\sigma^2}(x-\mu)^2\} dx = 1 \\ f = \int_{-\infty}^{\infty}exp\{-\frac{1}{2\sigma^2}(x-\mu)^2\} dx = \sqrt{2\pi\sigma^2} \\ \frac{df}{d \sigma} = \int_{-\infty}^{\infty}exp\{-\frac{1}{2\sigma^2}(x-\mu)^2\}(x-\mu)^2 \sigma^{-3} dx = \sqrt{2\pi} \\ \int_{-\infty}^{\infty} \frac{1}{\sqrt{2\pi\sigma^2}} exp\{-\frac{1}{2\sigma^2}(x-\mu)^2\}(x-\mu)^2 dx = \sigma^2 \\ \int_{-\infty}^{\infty}\mathcal{N}(x|\mu, \sigma^2) (x-\mu)^2 dx = \sigma^2 \\ \mathbb{E}[(x-\mu)^2] = var[x] = \sigma^2 \end{aligned}$$

   At last,

   $$\mathbb{E}[x^2] = var[x] + \mathbb{E}[x]^2 = \sigma^2 + \mu^2$$
9. (*)www Show that the mode (i.e. the maximum) of Gaussian distribution (1.46) is
   given by $\mu$, Similarly, show that the mode of the multivariate Gaussian
   $$\mathcal{N}(\boldsymbol{x}|\boldsymbol{\mu}, \boldsymbol{\Sigma}) = \frac{1}{(2\pi)^{D/2}}\frac{1}{|\boldsymbol{\Sigma}|^{1/2}}exp\{-\frac{1}{2}(\boldsymbol{x} - \boldsymbol{\mu})^T\boldsymbol{\Sigma^{-1}}(\boldsymbol{x} - \boldsymbol{\mu})\} \tag{1.52}$$

   is given by $\boldsymbol{\mu}$

   Solution:

   $$\frac{d}{dx}\mathcal{N} (x|\mu, \sigma^2) = \frac{1}{\sqrt{2\pi\sigma^2}}exp\{-\frac{1}{2\sigma^2}(x-\mu)^2\}\times (-(x-\mu)) = 0$$

   we got $x = \mu$

   the multivariate case: TODO

10. (*)www Suppose that the two variables $x$ and $z$ are statistically independent.
    Show that the mean and variance of their sum satisfies
    $$\mathbb{E}[x+z] = \mathbb{E}[x] + \mathbb{E}[z] \tag{1.128}$$
    $$var[x+z] = var[x] + var[z] \tag{1.129}$$

    Solution: if $x$ and $z$ are statistically independent, then $p(x, z) = p(x)p(z)$

    $$\begin{aligned} \mathbb{E}[x+z] &= \int\int(x+z)p(x,z)dxdz \\ &= \int\int(x+z)p(x)p(z)dxdz \\ &= \int\int xp(x)p(z)dxdz + \int\int zp(x)p(z)dxdz \\ &= \int(\int p(z)dz)xp(x)dx + \int(\int p(x)dx)zp(z)dz \\ &= \int xp(x)dx + \int zp(z) dz \\ &= \mathbb{E}[x] + \mathbb{E}[z] \end{aligned}$$
    $$\begin{aligned} var[x+z] &= \mathbb{E}[(x+z)^2] - \mathbb{E}[x+z]^2 \\ &= \mathbb{E}[x^2+2xz+z^2]- (\mathbb{E}[x] + \mathbb{E}[z])^2 \\ &= \mathbb{E}[x^2] + \mathbb{E}[2xz] + \mathbb{E}[z^2] - (\mathbb{E}[x]^2 + 2\mathbb{E}[x]\mathbb{E}[z]+\mathbb{E}[z]^2) \\ &= (\mathbb{E}[x^2] - \mathbb{E}[x]^2) + (\mathbb{E}[z^2]-\mathbb{E}[z]^2) + (\int\int 2xzp(x)p(z)dxdz - 2\mathbb{E}[x]\mathbb{E}[z]) \\ &= var[x] + var[z] + (2\int xp(x)dx \int zp(z)dz - 2\mathbb{E}[x]\mathbb{E}[z]) \\ &= var[x] + var[z] + (2\mathbb{E}[x]\mathbb{E}[z] - 2\mathbb{E}[x]\mathbb{E}[z]) \\ &= var[x] + var[z] \end{aligned}$$