Math Problems Vault

Problem 4.6.1

For the model (3.3.1), show that $\left.\delta^{\lambda }(x)\right|_{\lambda =\hat{\lambda }}=\delta^{\hat{\lambda }(x)}$ , where the Bayes estimator $\delta^{\lambda }(x)$ minimizes $\int L[\theta , d(x)] \pi (\theta \mid x, \lambda ) d \theta$ and the empirical Bayes estimator $\delta^{\hat{\lambda }(x)}$ minimizes $\int L[\theta , d(x)] \pi (\theta \mid x, \hat{\lambda }) d \theta$ .

?

Problem 4.6.2

(a)(b)

This problem will investigate conditions under which an empirical Bayes estimator is a Bayes estimator. Expression (6.6.3) is a true posterior expected loss if $\pi (\theta \mid \mathbf{x}, \hat{\lambda }(\mathbf{x}))$ is a true posterior.

From the hierarchy

\begin{align} & X \mid \theta \sim f(x \mid \theta ), \\ & \Theta \mid \lambda \sim \pi (\theta \mid \lambda ), \end{align}

define the joint distribution of $\mathbf{X}$ and $\Theta$ to be $(\mathbf{X}, \theta ) \sim g(\mathbf{x}, \theta )=f(\mathbf{x} \mid \theta ) \pi (\theta \mid \hat{\lambda }(\mathbf{x}))$ , where $\pi (\theta \mid \hat{\lambda }(\mathbf{x}))$ is obtained by substituting $\hat{\lambda }(\mathbf{x})$ for $\lambda$ in $\pi (\theta \mid \lambda )$ .

?

(a)

Show that, for this joint density, the formal Bayes estimator is equivalent to the empirical Bayes estimator from the hierarchical model.

(b)

If $f(\cdot \mid \theta )$ and $\pi (\cdot \mid \lambda )$ are proper densities, then $\int g(\mathbf{x}, \theta ) d \theta <\infty$ . However, $\iint g(\mathbf{x}, \theta ) d \mathbf{x} d \theta$ need not be finite.

Problem 4.6.3

For the model (6.3.1), the Bayes estimator $\delta^{\lambda }(x)$ minimizes $\int L(\theta , d(x)) \times \pi (\theta \mid x, \lambda ) d \theta$ and the empirical Bayes estimator, $\delta^{\hat{\lambda }}(x)$ , minimizes $\int L(\theta , d(x)) \pi (\theta \mid x, \hat{\lambda }(x)) d \theta$ . Show that $\left.\delta^{\lambda }(x)\right|_{\lambda =\hat{\lambda }(x)}=\delta^{\hat{\lambda }}(x)$ .

?

Problem 4.6.4

(a)(b)

For the situation of Example 6.1:

?

(a)

Show that

\int _{-\infty }^{\infty } e^{-n / 2 \sigma ^{2}(\bar{x}-\theta )^{2}} e^{(-1 / 2) \theta ^{2} / \tau ^{2}} d \theta =\sqrt{2 \pi }\left(\frac{\sigma ^{2} \tau ^{2}}{\sigma ^{2}+\tau ^{2}}\right)^{1 / 2} e^{(-n / 2) \bar{x}^{2} / \sigma ^{2}+n \tau ^{2}}

and, hence, establish (6.6.4).

(b)

Verify that the marginal MLE of $\sigma^{2}+n \tau^{2}$ is $n \bar{x}^{2}$ and that the empirical Bayes estimator is given by (6.6.5).

Problem 4.6.5

(a)(b)

Referring to Example 6.2:

?

(a)

Show that the Bayes risk, $r\left(\pi , \delta^{\pi }\right)$ , of the Bayes estimator (6.6.7) is given by

r\left(\pi , \delta ^{\pi }\right)=k \mathbb {E}\left[\operatorname {var}\left(p_{k} \mid x_{k}\right)\right]=\frac{k a b}{(a+b)(a+b+1)(a+b+n)}.

(b)

Show that the Bayes risk of the unbiased estimator $\mathbf{X} / n=\left(X_{1} / n, \ldots , X_{k} / n\right)$ is given by

r(\pi , \mathbf{X} / n)=\frac{k a b}{n(a+b+1)(a+b)}.

Problem 4.6.6

Theorem 6.3 states: for the situation of Corollary 3.3 (if $X=(X_{1}, \ldots , X_{p})$ has the density $p_{\eta }(x)=e^{\sum_{i} \eta_{i} x_{i}-A(\eta )} h(x)$ and $\eta$ has prior density $\pi (\eta )$ , the Bayes estimator of $\eta$ under squared error loss is $\mathbb {E}\left[\eta_{i} \mid x\right]=\frac{\partial }{\partial x_{i}} \log m(x)-\frac{\partial }{\partial x_{i}} \log h(x)$ ), with prior distribution $\pi (\eta \mid \lambda )$ , suppose $\hat{\lambda }(x)$ is the MLE of $\lambda$ based on $m(x \mid \lambda )$ . Then, the empirical Bayes estimator is

\mathbb {E}\left[\eta _{i} \mid x, \hat{\lambda }\right]=\frac{\partial }{\partial x_{i}} \log m(x \mid \hat{\lambda }(x))-\frac{\partial }{\partial x_{i}} \log h(x).

Extend Theorem 6.3 to the case of Theorem 3.2; that is, if $\mathbf{X}$ has density (3.3.7) and $\boldsymbol {\eta }$ has prior density $\pi (\boldsymbol {\eta } \mid \gamma )$ , then the empirical Bayes estimator is

\mathbb {E}\left[\left.\sum \eta _{i} \frac{\partial T_{i}(\mathbf{x})}{\partial x_{j}} \right\rvert \, \mathbf{x}, \hat{\gamma }(\mathbf{x})\right]=\frac{\partial }{\partial x_{j}} \log m(\mathbf{x} \mid \hat{\gamma }(\mathbf{x}))-\frac{\partial }{\partial x_{j}} \log h(\mathbf{x}),

where $m(\mathbf{x} \mid \gamma )$ is the marginal distribution of X and $\hat{\gamma }(\mathbf{x})$ is the marginal MLE of $\gamma$ .

?

Problem 4.6.7

(a)(b)

?

(a)

For $p_{\eta }(\mathbf{x})$ of (1.5.2), show that for any prior distribution $\pi (\eta \mid \lambda )$ that is dependent on a hyperparameter $\lambda$ , the empirical Bayes estimator is given by

\mathbb {E}\left[\left.\sum _{i=1}^{s} \eta _{i} \frac{\partial }{\partial x_{j}} T_{i}(\mathbf{x}) \right\rvert \, \mathbf{x}, \hat{\lambda }\right]=\frac{\partial }{\partial x_{j}} \log m_{\pi }(\mathbf{x} \mid \hat{\lambda }(\mathbf{x}))-\frac{\partial }{\partial x_{j}} \log h(\mathbf{x}).

where $m_{\pi }(\mathbf{x})=\int p_{\theta }(\mathbf{x}) \pi (\theta ) d \theta$ .

(b)

If X has the distribution $p_{\theta }(\mathbf{x})$ of (1.5.1), show that a similar formulas holds, that is,

\mathbb {E}\left[\mathcal{T} \eta (\theta ) \mid \hat{\lambda }\right]=\nabla \log m_{\pi }(\mathbf{x} \mid \hat{\lambda })-\nabla \log h(\mathbf{x}),

where $\mathcal{T}=\left\{ \partial T_{i} / \partial x_{j}\right\}$ is the Jacobian of $\mathcal{T}$ and $\nabla a$ is the gradient vector of $a$ , that is, $\nabla a=\left\{ \partial a / \partial x_{i}\right\}$ .

Problem 4.6.8

(a)(b)

For each of the following situations, write the empirical Bayes estimator of the natural parameter (under squared error loss) in the form (6.6.12), using the marginal likelihood estimator of the hyperparameter $\lambda$ . Evaluate the expressions as far as possible.

?

(a)

$X_{i} \sim N\left(0, \sigma_{i}^{2}\right), i=1, \ldots , p$ , independent; $1 / \sigma_{i}^{2} \sim \operatorname {Exponential}(\lambda )$ .

(b)

$X_{i} \sim N\left(\theta_{i}, 1\right), \quad i=1, \ldots , p$ , independent, $\theta_{i} \sim D E(0, \lambda )$ .

Problem 4.6.9

(a)(b)

Strawderman (1992) shows that the James-Stein estimator can be viewed as an empirical Bayes estimator in an arbitrary location family. Let $\mathbf{X}_{p \times 1} \sim f(x-\theta )$ , with $\mathbb {E}\left[\mathbf{X}\right]=\theta$ and var $\mathbf{X}=\sigma^{2} I$ . Let the prior be $\theta \sim f^{* n}$ , the $n$ -fold convolution of $f$ with itself. [The convolution of $f$ with itself is $f^{* 2}(x)=\int f(x-y) f(y) d y$ . The $n$ -fold convolution is $f^{* n}(x)=\int f *(n-1)(x)(x-y) f(y) d y$ .] Equivalently, let $U_{i} \sim f, i=0, \cdots , n$ , iid, $\theta =\sum_{1}^{n} U_{i}$ , and $\mathbf{X}=U_{0}+\theta$ .

?

(a)

Show that the Bayes rule against squared error loss is $\frac{n}{n+1} \mathbf{x}$ . Note that $n$ is a prior parameter.

(b)

Show that $|\mathbf{X}|^{2} /\left(p \sigma^{2}\right)$ is an unbiased estimator of $n+1$ , and hence that an empirical Bayes estimator of $\theta$ is given by $\delta^{E B}=\left[1-\left(p \sigma^{2} /|\mathbf{x}|^{2}\right)\right] \mathbf{x}$ .

Problem 4.6.10

(a)(b)(c)

Show for the hierarchy of Example 3.4, where $\sigma^{2}$ and $\tau^{2}$ are known but $\mu$ is unknown, that:

?

(a)

The empirical Bayes estimator of $\theta_{i}$ , based on the marginal MLE of $\theta_{i}$ , is $\frac{\tau^{2}}{\sigma^{2}+\tau^{2}} X_{i}+ \frac{\sigma^{2}}{\sigma^{2}+\tau^{2}} \bar{X}$ .

(b)

The Bayes risk, under sum-of-squared-errors loss, of the empirical Bayes estimator from part (a) is

p \sigma ^{2}-\frac{2(p-1)^{2} \sigma ^{4}}{p\left(\sigma ^{2}+\tau ^{2}\right)}+(p-1)\left(\frac{\sigma ^{2}}{\sigma ^{2}+\tau ^{2}}\right)^{2} \sum _{i=1}^{p} \mathbb {E}\left[(X_{i}-\bar{X})^{2}\right].

(c)

The minimum risk of the empirical Bayes estimator is attained when all $\theta_{i} \mathrm{~ s}$ are equal. [Hint: Show that $\sum_{i=1}^{p} \mathbb {E}\left[\left(X_{i}-\bar{X}\right)^{2}\right]=\sum_{i=1}^{p}\left(\theta_{i}-\bar{\theta }\right)^{2}+(p-1) \sigma^{2}$ .]

Problem 4.6.11

For $\mathbb {E}\left[\Theta \mid \mathbf{x}\right]$ of (5.5.8), show that as $v \rightarrow \infty , \mathbb {E}\left[\Theta \mid x\right] \rightarrow \left[p /\left(p+\sigma^{2}\right)\right] \bar{x}$ , the Bayes estimator under a $N(0,1)$ prior.

?

Problem 4.6.12

(a)(b)(c)

?

(a)

Show that the empirical Bayes $\delta^{E B}(\bar{x})=\left(1-\sigma^{2} / \max \left\{ \sigma^{2}, p \bar{x}^{2}\right\} \right) \bar{x}$ of (6.6.5) has bounded mean squared error.

(b)

Show that a variation of $\delta^{E B}(\bar{x})$ , of part (a), $\delta^{v}(\bar{x})=\left[1-\sigma^{2} /\left(v+p \bar{x}^{2}\right)\right] \bar{x}$ , also has bounded mean squared error.

(c)

For $\sigma^{2}=\tau^{2}=1$ , plot the risk functions of the estimators of parts (a) and (b).

[Thompson (1968a, 1968b) investigated the mean squared error properties of estimators like those in part (b). Although such estimators have smaller mean squared error than $\bar{x}$ for small values of $\theta$ , they always have larger mean squared error for larger values of $\theta$ .]

Problem 4.6.13

(a)(b)

?

(a)

For the hierarchy (5.5.7), with $\sigma^{2}=1$ and $p=10$ , evaluate the Bayes risk $r\left(\pi , \delta^{\pi }\right)$ of the Bayes estimator (5.5.8) for $v=2,5$ , and 10.

(b)

Calculate the Bayes risk of the estimator $\delta^{v}$ of Problem 6.12(b). Find a value of $v$ that yields a good approximation to the risk of the hierarchical Bayes estimator. Compare it to the Bayes risk of the empirical Bayes estimator of Problem 6.12(a).

Problem 4.6.14

Referring to Example 6.6, show that the empirical Bayes estimator is also a hierarchical Bayes estimator using the prior $\gamma (b)=1 / b$ .

?

Problem 4.6.15

(a)(b)(c)

The Taylor series approximation to the estimator (5.5.8) is carried out in a number of steps. Show that:

?

(a)

Using a first-order Taylor expansion around the point $\bar{x}$ , we have

\begin{align} \frac{1}{\left(1+\theta ^{2} / v\right)^{(v+1) / 2}}= & \frac{1}{\left(1+\bar{x}^{2} / v\right)^{(v+1) / 2}} \\ & -\frac{v+1}{v} \frac{\bar{x}}{\left(1+\bar{x}^{2} / v\right)^{(v+3) / 2}}(\theta -\bar{x})+R(\theta -\bar{x}) \end{align}

where the remainder, $R(\theta -\bar{x})$ , satisfies $R(\theta -\bar{x}) /(\theta -\bar{x})^{2} \rightarrow 0$ as $\theta \rightarrow \bar{x}$ .

(b)

The remainder in part (a) also satisfies

\int _{-\infty }^{\infty } R(\theta -\bar{x}) e^{-\frac{p}{2 \sigma ^{2}}(\theta -\bar{x})^{2}} d \theta =O\left(1 / p^{3 / 2}\right)

(c)

The numerator and denominator of (5.5.8) can be written

\int _{-\infty }^{\infty } \frac{1}{\left(1+\theta ^{2} / \nu \right)^{(\nu +1) / 2}} e^{-\frac{p}{2 \sigma ^{2}}(\theta -\bar{x})^{2}} d \theta =\frac{\sqrt{2 \pi \sigma ^{2} / p}}{\left(1+\bar{x}^{2} / \nu \right)^{(\nu +1) / 2}}+O\left(\frac{1}{p^{3 / 2}}\right)

and

\begin{align} \int _{-\infty }^{\infty } & \frac{\theta }{\left(1+\theta ^{2} / v\right)^{(v+1) / 2}} e^{-\frac{p}{2 \sigma ^{2}}(\theta -\bar{x})^{2}} d \theta \\ & =\frac{\sqrt{2 \pi \sigma ^{2} / p}}{\left(1+\bar{x}^{2} / v\right)^{(v+1) / 2}}\left[1-\frac{(v+1) / v}{\left(1+\bar{x}^{2} / v\right)}\right] \bar{x}+O\left(\frac{1}{p^{3 / 2}}\right) \end{align}

which yields (5.6.32).

Problem 4.6.16

(a)(b)

For the situation of Example 6.7:

?

(a)

Calculate the values of the approximation (5.6.32) for the values of Table 6.2. Are there situations where the estimator (5.6.32) is clearly preferred over the empirical Bayes estimator (5.6.5) as an approximation to the hierarchical Bayes estimator (5.5.8)?

(b)

Extend the argument of Problem 6.15 to calculate the next term in the expansion and, hence, obtain a more accurate approximation to the hierarchical Bayes estimator (5.5.8). For the values of Table 6.2, is this new approximation to (5.5.8) preferable to (5.6.5) and (5.6.32)?

Problem 4.6.17

(a)(b)(c)(d)

?

(a)

Show that if $b(\cdot )$ has a bounded second derivative, then

\int b(\lambda ) e^{-n h(\lambda )} d \lambda =b(\hat{\lambda }) \sqrt{\frac{2 \pi }{n h^{\prime \prime }(\hat{\lambda })}} e^{-n h(\hat{\lambda })}+O\left(\frac{1}{n^{3 / 2}}\right)

where $h(\hat{\lambda })$ is the unique minimum of $h(\lambda ), h^{\prime \prime }(\lambda ) \neq 0$ , and $n h(\hat{\lambda }) \rightarrow$ constant as $n \rightarrow \infty$ . [Hint: Expand both $b(\cdot )$ and $h(\cdot )$ in Taylor series around $\hat{\lambda }$ , up to second-order terms. Then, do the term-by-term integration.] This is the Laplace approximation for an integral. For refinements and other developments of this approximation in Bayesian inference, see Tierney and Kadane 1986, Tierney, Kass, and Kadane 1989, and Robert 1994a (Section 9.2.3).

(b)

For the hierarchical model (5.5.1), the posterior mean can be approximated by

\mathbb {E}\left[\Theta \mid x\right]=e^{-n h(\hat{\lambda })}\left[\frac{2 \pi }{n h^{\prime \prime }(\hat{\lambda })}\right]^{1 / 2} \mathbb {E}\left[\Theta \mid x, \hat{\lambda }\right]+O\left(\frac{1}{n^{3 / 2}}\right)

where $h=\frac{1}{n} \log \pi (\lambda \mid x)$ and $\hat{\lambda }$ is the mode of $\pi (\lambda \mid x)$ , the posterior distribution of $\lambda$ .

(c)

If $\pi (\lambda \mid x)$ is the normal distribution with mean $\hat{\lambda }$ and variance $\sigma^{2}=\left[-\left(\partial^{2} / \partial \lambda^{2}\right) \times \left.\log \pi (\lambda \mid x)\right|_{\lambda =\hat{\lambda }}\right]^{-1}$ , then $\mathbb {E}\left[\Theta \mid x\right]=\mathbb {E}\left[\Theta \mid x, \hat{\lambda }\right]+O\left(1 / n^{3 / 2}\right)$ .

(d)

Show that the situation in part (c) arises from the hierarchy

\begin{align} X_{i} \mid \theta _{i} & \sim N\left(\theta _{i}, \sigma ^{2}\right), \\ \theta _{i} \mid \lambda & \sim N\left(\lambda , \tau ^{2}\right), \\ \lambda & \sim \text{ Uniform }(-\infty , \infty ). \end{align}

Problem 4.6.18

(a)(b)

?

(a)

Apply the Laplace approximation (5.6.33) to obtain an approximation to the hierarchical Bayes estimator of Example 6.6.

(b)

Compare the approximation from part (a) with the empirical Bayes estimator (5.6.24). Which is a better approximation to the hierarchical Bayes estimator?

Problem 4.6.19

Apply the Laplace approximation (5.6.33) to the hierarchy of Example 6.7 and show that the resulting approximation to the hierarchical Bayes estimator is given by (5.6.32).

?

Problem 4.6.20

(a)(b)(c)

?

(a)

Verify (6.6.37), that under squared error loss

r(\pi , \delta )=r\left(\pi , \delta ^{\pi }\right)+\mathbb {E}\left[(\delta -\delta ^{\pi })^{2}\right].

(b)

For $X \sim \operatorname {binomial}(p, n), L(p, \delta )=(p-\delta )^{2}$ , and $\pi =\left\{ \pi : \pi =\operatorname {beta}(a, b), a>0\text{, }b>0\right\}$ , determine whether $\hat{p}=x / n$ or $\delta^{0}=\left(a_{0}+x\right) /\left(a_{0}+b_{0}+n\right)$ is more robust, according to (6.6.37).

(c)

Is there an estimator of the form $(c+x) /(c+d+n)$ that you would consider more robust, in the sense of (6.6.37), than either estimator in part (b)?

[In part (b), for fixed $n$ and $\left(a_{0}, b_{0}\right)$ , calculate the Bayes risk of $\hat{p}$ and $\delta^{0}$ for a number of $(a, b)$ pairs.]

Problem 4.6.21

(a)(b)

?

(a)

Establish (6.6.39) and (6.6.40) for the class of priors given by (6.6.38).

(b)

Show that the Bayes estimator based on $\pi (\theta ) \in \pi$ in (6.6.38), under squared error loss, is given by (6.6.41).