Estimation

Consider the joint probability density $f_{X,Z}(x,z) = e^{-z}$ for $0 \leq x \leq z$ and $f_{X,Z}(x,z) = 0$ otherwise. Find the pair $x,z$ of values that maximize this density. Find the marginal density $f_Z(z)$ and find the value of $z$ that maximizes this.

Let $f_{X,Z,Y}(x,z,y)$ be $y^2 e^{-yz}$ for $0 \leq x \leq z$, $1 \leq y \leq 2$ and be $0$ otherwise. Conditional on an observation $Y = y$, find the joint MAP estimate of $X, Z$. Find $f_{Z \mid Y}(z \mid y)$, the marginal density of $Z$ conditional on $Y = y$, and find the MAP estimate of $Z$ conditional on $Y = y$.

Explain (using the results of Chapter 3) why $Y$ and $U$ are jointly Gaussian and why they are statistically independent.

Without using any matrices, write down the joint probability density of $Y$ and $U$. Verify your answer from (3.22).

Find the MMSE estimate $\hat{x}(y)$ of $X$ conditional on a given sample value $y$ of $Y$. You can derive this from first principles, or use (10.9) or Example 10.2.2.

Show that the estimation error $\xi = X-\hat{X}(Y)$ is equal to $-U/2$.

Write down the probability density of $U$ conditional on $Y = y$ and that of $\xi$ conditional on $X = y$.

Draw a sketch in the $x,z$ plane of the equiprobability contours of $X$ and $Z$. Explain why these are also equiprobability contours for $Y$ and $U$. For some given sample value of $Y$, say $Y = 1$, illustrate the set of points for which $x+z = 1$. For a given point on this line, illustrate the sample value of $U$.

Let $X, Z_1, Z_2, \ldots , Z_n$ be independent zero-mean Gaussian rv's with variances $\sigma_X^2, \sigma_{Z_1}^2, \ldots \sigma_{Z_n}^2$ respectively. Let $Y_j = h_j X+Z_j$ for $j \geq 1$ and let $\overrightarrow {Y} = (Y_1, \ldots Y_n)^{\mathrm{T}}$. Use (10.9) to show that the MMSE estimate of $X$ conditional on $\overrightarrow {Y} = \overrightarrow {y} = (y_1, \ldots , y_n)^{\mathrm{T}}$ is given by

Hint: Let the row vector $\overrightarrow {g}^{\mathrm{T}}$ be $[K_{X \cdot Y}][K_Y^{-1}]$ and multiply $\overrightarrow {g}^{\mathrm{T}}$ by $K_Y$ to solve for $\overrightarrow {g}^{\mathrm{T}}$.

Let $\xi = X-\hat{X}(\overrightarrow {Y})$ and show that (10.29) is valid, i.e., that

Show that (10.28), i.e., $\hat{x}(\overrightarrow {y}) = \sigma_{\xi }^2 \sum_{j=1}^{n} h_j y_j/\sigma_{Z_j}^2$, is valid.

Show that the expression in (10.29) is equivalent to the iterative expression in (10.27).

Show that the expression in (10.28) is equivalent to the iterative expression in (10.25).

Write out the MMSE estimate and variance of the estimation error from (10.28) and (10.29) where each $h_i = 1$.

For arbitrary numbers $h_1, \ldots , h_n$, consider the equations $h_i Y_i = h_i X+h_i Z_i$. Let $U_i = h_i Y_i$. Write out the MMSE estimate of $X$ as a function of $U_1, \ldots , U_n$, still using the rv's $Z_1, \ldots , Z_n$. Hint: Note that if you scale each observation $U_i$ to $U_i/h_i$ you have the same estimation problem as in (a).

Now let $W_i = h_i Z_i$. Write out the same MMSE estimate in terms of both $U_i, \ldots , U_n$ and $W_1, \ldots , W_n$ and show that you now have (10.28) with all the $h_i$'s back in the equation. Hint: You still have the basic equation in (a), but the variance of the $Z_i$'s must be expressed in terms of the $W_i$'s. Write out the error variance from (10.29) in the same way.

Assume that $X_1 \sim \mathscr {N}\left(\overline{X}_1, \sigma_{X_1}^2\right)$ and that for each $n \geq 1$, $X_{n+1} = \alpha X_n+W_n$, where $0 < \alpha < 1$, $W_n \sim \mathscr {N}\left(0, \sigma_W^2\right)$, and $X_1, W_1, W_2, \ldots$, are independent. Show that for each $n \geq 1$,

Show directly, by comparing the equation $\sigma_{X_n}^2 = \alpha^2 \sigma_{X_{n-1}}^2+\sigma_W^2$ for each two adjoining values of $n$, that $\sigma_{X_n}^2$ moves monotonically from $\sigma_{X_1}^2$ to $\sigma_W^2/(1-\alpha^2)$ as $n \rightarrow \infty$.

Assume that sample values of $Y_1, Y_2, \ldots$, are observed, where $Y_n = h X_n+Z_n$ and where $Z_1, Z_2, \ldots$, are IID zero-mean Gaussian rv's with variance $\sigma_Z^2$. Assume that $Z_1, \ldots , W_1, \ldots , X_1$ are independent and assume $h \geq 0$. Rewrite the recursion for the variance of the estimation error in (10.41) for this special case. Show that if $h/\sigma_Z = 0$, then $\sigma_{\xi_n}^2 = \sigma_{X_n}^2$ for each $n \geq 1$. Hint: Compare the recursion in (b) to that for $\sigma_{\xi_n}^2$.

Show from the recursion that $\sigma_{\xi_n}^2$ is a decreasing function of $h/\sigma_Z$ for each $n \geq 2$. Use this to show that $\sigma_{\xi_n}^2 \leq \sigma_{X_n}^2$ for each $n$. Explain (without equations) why this result must be true.

Show that the sequence $\left\{ \sigma_{\xi_n}^2; n \geq 1\right\}$ is monotonic in $n$. Hint: Use the same technique as in (b). From this and (d), show that $\lambda = \lim_{n \rightarrow \infty } \sigma_{\xi_n}^2$ exists. Show that the limit satisfies (10.42) (note that (10.42) must have $2$ roots, one positive and one negative, so the limit must be the positive root).

Show that for each $n \geq 1$, $\sigma_{\xi_n}^2$ is increasing in $\sigma_W^2$ and increasing in $\alpha$. Note: This increase with $\alpha$ is surprising, since when $\alpha$ is close to one, $X_n$ changes slowly, so we would expect to be able to track $X_n$ well. The problem is that $\lim_n \sigma_{X_n}^2 = \sigma_W^2/(1-\alpha^2)$ so the variance of the untracked $X_n$ is increasing without limit as $\alpha$ approaches $1$. Part (g) is somewhat messy, but resolves this issue.

Show that if the recursion is expressed in terms of $\beta = \sigma_W^2/(1-\alpha^2)$ and $\alpha$, then $\lambda$ is decreasing in $\alpha$ for constant $\beta$.

Assume that $\overrightarrow {X}$ and $\overrightarrow {Y}$ are zero-mean, jointly Gaussian, jointly non-singular, and related by $\overrightarrow {Y} = [H]\overrightarrow {X}+\overrightarrow {Z}$. Show that the MMSE estimate can be expressed as

Hint: Recall from (10.9) that $\hat{\overrightarrow {x}}(\overrightarrow {y}) = [G]\overrightarrow {y}$. Relate $[G]$ and $[H]$ by using (3.115).

Show that

Hint: A reasonable approach is to start with (10.10), i.e., $[K_{\xi }] = [K_X]-[K_{X \cdot Y} K_Y^{-1} K_{X \cdot Y}^{\mathrm{T}}]$. Premultiply both sides by $[K_X^{-1}]$ and post-multiply both sides by $[K_{\xi }^{-1}]$. The final product of matrices here can be expressed as $[H^{\mathrm{T}} K_Z^{-1} H]$ by using the hint in (a) on $[K_Z^{-1} H]$.

Write out $\mathbb {E}\left[(X-\overrightarrow {g}^{\mathrm{T}} \overrightarrow {Y})^2\right] = \sigma_X^2-2[K_{X \cdot Y}]\overrightarrow {g}+\overrightarrow {g}^{\mathrm{T}}[K_Y]\overrightarrow {g}$ as a function of $\overrightarrow {g} = (g_1, g_2, \ldots , g_n)^{\mathrm{T}}$ and take the partial derivative of the function with respect to $g_i$ for each $i$, $1 \leq i \leq n$. Show that the vector of these partial derivatives is $-2[K_{X \cdot Y}]+2\overrightarrow {g}^{\mathrm{T}}[K_Y]$.

Explain why the stationary point here is actually a minimum.

Show that

Show that

and find the $B, C$ for which this is true.

Show that $[K_X] = 2([K_{\mathrm{re}}]-[K_{\mathrm{ri}}])$.

Show that $[K_X^{-1}] = \frac{1}{2}(B-iC)$.

Define $f_X(\overrightarrow {x}) = f_U(\overrightarrow {u})$ for $\overrightarrow {u} = (\overrightarrow {x}_{\mathrm{re}}^{\mathrm{T}}, \overrightarrow {x}_{\mathrm{im}}^{\mathrm{T}})^{\mathrm{T}}$ and show that

Show that

Hint: Recall that elementary row operations do not change the value of a determinant.

Show that

Hint: Recall that elementary column operations do not change the value of a determinant.

Show that

and from this conclude that (3.108) is valid.

Let $X$ be a circularly-symmetric, zero-mean, complex Gaussian rv with variance $1$. Show that

Recall that the real part and imaginary part each have variance $1/2$.

Let $\overrightarrow {X}$ be an $n$-dimensional, circularly-symmetric, complex Gaussian zero-mean random vector with $K_X = [I_n]$. Show that

Let $\overrightarrow {Y} = [H]\overrightarrow {X}$ where $[H]$ is $n \times n$ and invertible. Show that

where $v$ is $d\overrightarrow {y}/d\overrightarrow {x}$, the ratio of an incremental $2n$-dimensional volume element after being transformed by $[H]$ to that before being transformed.

Use this to show that (3.108) is valid.

Let $Y = X^2+Z$, where $Z$ is a zero-mean unit variance Gaussian rv. Show that no unbiased estimate of $X$ exists from observation of $Y$.

Let $Y = X+Z$, where $Z$ is uniform over $(-1,1)$ and $X$ is a parameter lying in $(-1,1)$. Show that $\hat{x}(y) = y$ is an unbiased estimate of $x$. Find a biased estimate $\hat{x}_1(y)$ for which $\left|\hat{x}_1(y)-x\right| \leq \left|\hat{x}(y)-x\right|$ for all $x$ and $y$ with strict inequality with positive probability for all $x \in (-1,1)$.

Assume that for each parameter value $x$, $Y$ is Gaussian, $\mathscr {N}\left(x, \sigma_Z^2\right)$. Show that $v_x(y)$ as defined in (10.103) is equal to $(y-x)/\sigma_Z^2$ and show that the Fisher information is equal to $1/\sigma_Z^2$.

Show that for ML estimation, the bias is $0$ for all $x$. Show that the Cramer--Rao bound is satisfied with equality for all $x$ in this case.

Consider using the MMSE estimator for the a priori distribution $X \sim \mathscr {N}\left(0, \sigma_X^2\right)$. Show that the bias satisfies $b_{\hat{x}}(x) = -x\sigma_Z^2/(\sigma_Z^2+\sigma_X^2)$.

Show that the MMSE estimator in (c) satisfies the Cramer--Rao bound with equality for each $x$. Note that the mean-squared error, as a function of $x$, is smaller than that for the ML case for small $x$ and larger for large $x$.