In Example 1.5, if $p$ has the improper prior density $\frac{1}{p(1-p)}$ , show that the posterior density of $p$ given $x$ is proper, provided $0<x<n$ .

Origin: Stated according to the original source

Problem 4.1.4

In Example 1.5, find the Jeffreys prior for $p$ and the associated Bayes estimator $\delta_{\Lambda }$ .

Origin: Stated according to the original source

Problem 4.1.5

(a)(b)

For the estimator $\delta_{\Lambda }$ of Problem 1.4,

Origin: Stated according to the original source

(a)

calculate the bias and maximum bias;

(b)

calculate the expected squared error and compare it with that of the UMVU estimator.

Problem 4.1.6

In Example 1.5, find the Bayes estimator $\delta$ of $p(1-p)$ when $p$ has the prior $B(a, b)$ .

Origin: Stated according to the original source

Problem 4.1.7

(a)(b)

For the situation of Example 1.5, the UMVU estimator of $p(1-p)$ is $\delta^{\prime }=[x(x- 1)] /[n(n-1)]$ (see Example 2.3.1 and Problem 2.3.1).

Origin: Stated according to the original source

(a)

Compare the estimator $\delta$ of Problem 1.6 with the UMVU estimator $\delta^{\prime }$ .

(b)

Compare the expected squared error of the estimator of $p(1-p)$ for the Jeffreys prior in Example 1.5 with that of $\delta^{\prime }$ .

Problem 4.1.8

In analogy with Problem 1.2, determine the possible shapes of the gamma density $\Gamma (g, 1 / \alpha ), \alpha , g>0$ .

Origin: Stated according to the original source

Problem 4.1.9

(a)(b)

Let $X_{1}, \ldots , X_{n}$ be iid according to the Poisson distribution $P(\lambda )$ and let $\lambda$ have a gamma distribution $\Gamma (g, \alpha )$ .

Origin: Stated according to the original source

(a)

For squared error loss, show that the Bayes estimator $\delta_{\alpha , g}$ of $\lambda$ has a representation analogous to (1.1.13).

(b)

What happens to $\delta_{\alpha , g}$ as (i) $n \rightarrow \infty$ , (ii) $\alpha \rightarrow \infty , g \rightarrow 0$ , or both?

Problem 4.1.10

For the situation of the preceding problem, solve the two parts corresponding to Problem 1.5(a) and (b).

Origin: Stated according to the original source

Problem 4.1.11

In Problem 1.9, if $\lambda$ has the improper prior density $d \lambda / \lambda$ (corresponding to $\alpha =g=$ 0 ), under what circumstances is the posterior distribution proper?

Origin: Stated according to the original source

Problem 4.1.12

(a)(b)

Solve the problems analogous to Problems 1.9 and 1.10 when the observations consist of a single random variable $X$ having a negative binomial distribution $N b(p, m)$ , $p$ has the beta prior $B(a, b)$ , and the estimand is

Origin: Stated according to the original source

(a)

$p$ and

(b)

$1 / p$ .

Section 4.2First Examples[16/0%]

Problem 4.2.1

(a)(b)

Referring to Example 1.5, suppose that $X$ has the binomial distribution $b(p, n)$ and the family of prior distributions for $p$ is the family of beta distributions $B(a, b)$ .

Origin: Stated according to the original source

(a)

Show that the marginal distribution of $X$ is the beta-binomial distribution with mass function

\binom {n}{x} \frac{\Gamma (a+b)}{\Gamma (a) \Gamma (b)} \frac{\Gamma (x+a) \Gamma (n-x+b)}{\Gamma (n+a+b)}.

(b)

Show that the mean and variance of the beta-binomial is given by

\mathbb {E}\left[X\right]=\frac{n a}{a+b} \quad \text{ and } \quad \text{ var } X=n\left(\frac{a}{a+b}\right)\left(\frac{b}{a+b}\right)\left(\frac{a+b+n}{a+b+1}\right).

[Hint: For part (b), the identities $\mathbb {E}\left[X\right]=E[\mathbb {E}\left[X \mid p\right]]$ and $\operatorname {var} X=\operatorname {var}[\mathbb {E}\left[X \mid p\right]]+ E[\operatorname {var}(X \mid p)]$ are helpful.]

Problem 4.2.2

(a)(b)(c)

For the situation of Example 2.1, Lindley and Phillips (1976) give a detailed account of the effect of stopping rules, which we can illustrate as follows. Let $X$ be the number of successes in $n$ Bernoulli trials with success probability $p$ .

Origin: Stated according to the original source

(a)

Suppose that the number of Bernoulli trials performed is a prespecifed number $n$ , so that we have the binomial sampling model, $\mathbb {P}\left(X=x\right)=\binom {n}{x} p^{x}(1-p)^{n-x}$ , $x=0,1, \ldots , n$ . Calculate the Bayes risk of the Bayes estimator (1.1.12) and the UMVU estimator of $p$ .

(b)

Suppose that the number of Bernoulli trials performed is a random variable $N$ . The value $N=n$ was obtained when a prespecified number, $x$ , of successes was observed so that we have the negative binomial sample model, $\mathbb {P}\left(N=n\right)=\binom {n-1}{x-1} p^{x}(1- p)^{n-x}$ , $n=x$ . Calculate the Bayes risk of the Bayes estimator and the UMVU estimator of $p$ .

(c)

Calculate the mean squared errors of all three estimators under each model. If it is unknown which sampling mechanism generated the data, which estimator do you prefer overall?

Problem 4.2.3

(a)(b)(c)

Show that the estimator (2.2.4) tends in probability

Origin: Stated according to the original source

(a)

to $\theta$ as $n \rightarrow \infty$ ,

(b)

to $\mu$ as $b \rightarrow 0$ , and

(c)

to $\theta$ as $b \rightarrow \infty$ .

Problem 4.2.4

(a)(b)(c)

Bickel and Mallows (1988) further investigate the relationship between unbiasedness and Bayes, specifying conditions under which these properties cannot hold simultaneously. In addition, they show that if a prior distribution is improper, then a posterior mean can be unbiased. Let $X \sim \frac{1}{\theta } f(x / \theta ), x>0$ , where $\int_{0}^{\infty } t f(t) d t=1$ , and let $\pi (\theta )=\frac{1}{\theta^{2}} d \theta$ , $\theta >0$ .

Origin: Stated according to the original source

(a)

Show that $\mathbb {E}\left[X \mid \theta \right]=\theta$ , so $X$ is unbiased.

(b)

Show that $\pi (\theta \mid x)=\frac{x^{2}}{\theta^{3}} f(x / \theta )$ is a proper density.

(c)

Show that $\mathbb {E}\left[\theta \mid x\right]=x$ , and hence the posterior mean, is unbiased.

Problem 4.2.5

DasGupta (1994) presents an identity relating the Bayes risk to bias, which illustrates that a small bias can help achieve a small Bayes risk. Let $X \sim f(x \mid \theta )$ and $\theta \sim \pi (\theta )$ . The Bayes estimator under squared error loss is $\delta^{\pi }=\mathbb {E}\left[\theta \mid x\right]$ . Show that the Bayes risk of $\delta^{\pi }$ can be written

r\left(\pi , \delta ^{\pi }\right)=\int _{\Theta } \int _{\mathcal{X}}\left[\theta -\delta ^{\pi }(x)\right]^{2} f(x \mid \theta ) \pi (\theta ) d x d \theta =\int _{\Theta } \theta b(\theta ) \pi (\theta ) d \theta

where $b(\theta )=\mathbb {E}\left[\delta^{\pi }(X) \mid \theta \right]-\theta$ is the bias of $\delta^{\pi }$ .

Origin: Stated according to the original source

Problem 4.2.6

Verify the estimator (2.2.10).

Origin: Stated according to the original source

Problem 4.2.7

In Example 2.6, verify that the posterior distribution of $\tau$ is $\Gamma (r+g-1 / 2,1 /(\alpha +z))$ .

Origin: Stated according to the original source

Problem 4.2.8

In Example 2.6 with $\alpha =g=0$ , show that the posterior distribution given the $X$ 's of $\sqrt{n}(\theta -\bar{X}) / \sqrt{Z /(n-1)}$ is Student's $t$ -distribution with $n-1$ degrees of freedom.

Origin: Stated according to the original source

Problem 4.2.9

In Example 2.6, show that the posterior distribution of $\theta$ is symmetric about $\bar{x}$ when the joint prior of $\theta$ and $\sigma$ is of the form $h(\sigma ) d \sigma d \theta$ , where $h$ is an arbitrary probability density on $(0, \infty )$ .

Origin: Stated according to the original source

Problem 4.2.10

Rukhin (1978) investigates the situation when the Bayes estimator is the same for every loss function in a certain set of loss functions, calling such estimators universal Bayes estimators. For the case of Example 2.6, using the prior of the form of Problem 2.9, show that $\bar{X}$ is the Bayes estimator under every even loss function.

Origin: Stated according to the original source

Problem 4.2.11

(a)(b)

Let $X$ and $Y$ be independently distributed according to distributions $P_{\xi }$ and $Q_{\eta }$ , respectively. Suppose that $\xi$ and $\eta$ are real-valued and independent according to some prior distributions $\Lambda$ and $\Lambda^{\prime }$ . If, with squared error loss, $\delta_{\Lambda }$ is the Bayes estimator of $\xi$ on the basis of $X$ , and $\delta_{\Lambda^{\prime }}^{\prime }$ is that of $\eta$ on the basis of $Y$ ,

Origin: Stated according to the original source

(a)

show that $\delta_{\Lambda^{\prime }}^{\prime }-\delta_{\Lambda }$ is the Bayes estimator of $\eta -\xi$ on the basis of $(X, Y)$ ;

(b)

if $\eta >0$ and $\delta_{\Lambda^{\prime }}^{*}$ is the Bayes estimator of $1 / \eta$ on the basis of $Y$ , show that $\delta_{\Lambda } \cdot \delta_{\Lambda^{\prime }}^{*}$ is the Bayes estimator of $\xi / \eta$ on the basis of $(X, Y)$ .

Problem 4.2.12

For the density (2.2.13) and improper prior $(d \sigma / \sigma ) \cdot \left(d \sigma_{A} / \sigma_{A}\right)$ , show that the posterior distribution of $\left(\sigma , \sigma_{A}\right)$ continues to be improper.

Origin: Stated according to the original source

Problem 4.2.13

(a)(b)

Origin: Stated according to the original source

(a)

In Example 2.7, obtain the Jeffreys prior distribution of $(\sigma , \tau )$ .

(b)

Show that for the prior of part (a), the posterior distribution of $(\sigma , \tau )$ is proper.

Problem 4.2.14

Verify the Bayes estimator (2.2.14).

Origin: Stated according to the original source

Problem 4.2.15

(a)(b)(c)

Let $X \sim N(\theta , 1)$ and $L(\theta , \delta )=(\theta -\delta )^{2}$ .

Origin: Stated according to the original source

(a)

Show that $X$ is the limit of the Bayes estimators $\delta^{\pi_{n}}$ , where $\pi_{n}$ is $N(0,1)$ . Hence, $X$ is both generalized Bayes and a limit of Bayes estimators.

(b)

For the prior measure $\pi (\theta )=e^{a \theta }, a>0$ , show that the generalized Bayes estimator is $X+a$ .

(c)

For $a>0$ , show that there is no sequence of proper priors for which $\delta^{\pi_{n}} \rightarrow X+a$ .

This example is due to Farrell; see Kiefer 1966. Heath and Sudderth (1989), building on the work of Stone (1976), showed that inferences from this model are incoherent, and established when generalized Bayes estimators will lead to coherent (that is, noncontradictory) inferences. Their work is connected to the theory of "approximable by proper priors," developed by Stein (1965) and Stone (1965, 1970, 1976), which shows when generalized Bayes estimators can be looked upon as Bayes estimators.

Problem 4.2.16

(a)(b)

Origin: Stated according to the original source

(a)

For the situation of Example 2.8, verify that $\delta (x)=x / n$ is a generalized Bayes estimator.

(b)

If $X \sim N(0,1)$ and $L(\theta , \delta )=(\theta -\delta )^{2}$ , show that $X$ is generalized Bayes under the improper prior $\pi (\theta )=1$ .

Section 4.3Single-Prior Bayes[12/0%]

Problem 4.3.1

(a)(b)

For the situation of Example 3.1:

Origin: Stated according to the original source

(a)

Verify that the Bayes estimator will only depend on the data through $Y=\max_{i} X_{i}$ .

(b)

Show that $\mathbb {E}\left[\Theta \mid y, a, b\right]$ can be expressed as

\mathbb {E}\left[\Theta \mid y, a, b\right]=\frac{1}{b(n+a-1)} \frac{P\left(\chi _{2(n+a-1)}^{2}<2 / b y\right)}{P\left(\chi _{2(n+a)}^{2}<2 / b y\right)}

where $\chi_{v}^{2}$ is a chi-squared random variable with $v$ degrees of freedom. (In this form, the estimator is particularly easy to calculate, as many computer packages will have the chi-squared distribution built in.)

Problem 4.3.2

(a)(b)(c)

Let $X_{1}, \ldots , X_{n}$ be iid from $\operatorname {Gamma}(a, b)$ where $a$ is known.

Origin: Stated according to the original source

(a)

Verify that the conjugate prior for the natural parameter $\eta =-1 / b$ is equivalent to an inverted gamma prior on $b$ .

(b)

Using the prior in part (a), find the Bayes estimator under the losses (i) $L(b, \delta )= (b-\delta )^{2}$ and (ii) $L(b, \delta )=(1-\delta / b)^{2}$ .

(c)

Express the estimator in part (b)(i) in the form (3.3.9). Can the same be done for the estimator in part (b)(ii)?

Problem 4.3.3

(a)(b)

Corollary 3.3 states: if $X=(X_{1}, \ldots , X_{p})$ has the density $p_{\eta }(x)=e^{\sum_{i=1}^{p} \eta_{i} x_{i}-A(\eta )} h(x)$ and $\eta$ has prior density $\pi (\eta )$ , the Bayes estimator of $\eta$ under the loss $L(\eta , \delta )=\sum (\eta_{i}-\delta_{i})^{2}$ is given by

\mathbb {E}\left[\eta _{i} \mid x\right]=\frac{\partial }{\partial x_{i}} \log m(x)-\frac{\partial }{\partial x_{i}} \log h(x).

Origin: Stated according to the original source

(a)

Prove Corollary 3.3.

(b)

Verify the calculation of the Bayes estimator in Example 3.4.

Problem 4.3.4

Using Stein's identity (Lemma 1.5.15), show that if $X_{i} \sim p_{\eta_{i}}(x)$ of (3.3.7), then

\begin{align} \mathbb {E}_{\eta }\left[-\nabla \log h(\mathbf{X})\right] & =\boldsymbol {\eta }, \\ R(\boldsymbol {\eta },-\nabla \log h(\mathbf{X})) & =\sum _{i=1}^{p} \mathbb {E}_{\eta }\left[-\frac{\partial ^{2}}{\partial X_{i}^{2}} \log h(\mathbf{X})\right]. \end{align}

Origin: Stated according to the original source

Problem 4.3.5

(a)(b)(c)

Origin: Stated according to the original source

(a)

If $X_{i} \sim \operatorname {Gamma}(a, b), i=1, \ldots , p$ , independent with $a$ known, calculate $-\nabla \log h(\mathbf{x})$ and its expected value.

(b)

Apply the results of part (a) to the situation where $X_{i} \sim N\left(0, \sigma_{i}^{2}\right), i=1, \ldots , p$ , independent. Does it lead to an unbiased estimator of $\sigma_{i}^{2}$ ? [Note: For part (b), squared error loss on the natural parameter $1 / \sigma^{2}$ leads to the loss $L\left(\sigma^{2}, \delta \right)=\left(\sigma^{2} \delta -1\right)^{2} / \sigma^{4}$ for estimation of $\sigma^{2}$ .]

(c)

X_{i} \sim \frac{\tan \left(a_{i} \pi \right)}{\pi } x^{a_{i}}(1-x)^{-1}, 0<x<1, \quad i=1, \ldots , p, \text{ independent, }

evaluate $-\nabla \log h(\mathbf{X})$ and show that it is an unbiased estimator of $\mathbf{a}=\left(a_{1}, \ldots , a_{p}\right)$ .

Problem 4.3.6

(a)(b)(c)

For the situation of Example 3.6:

Origin: Stated according to the original source

(a)

Show that if $\delta$ is a Bayes estimator of $\theta$ , then $\delta^{\prime }=\delta / \sigma^{2}$ is a Bayes estimator of $\eta$ , and hence $R(\theta , \delta )=\sigma^{4} R\left(\eta , \delta^{\prime }\right)$ .

(b)

Show that the risk of the Bayes estimator of $\eta$ is given by

\frac{p \tau ^{4}}{\sigma ^{2}\left(\sigma ^{2}+\tau ^{2}\right)^{2}}+\left(\frac{\sigma ^{2}}{\sigma ^{2}+\tau ^{2}}\right)^{2} \sum a_{i}^{2}

where $a_{i}=\eta_{i}-\mu / \sigma^{2}$ .

(c)

If $\Sigma a_{i}^{2}=k$ , a fixed constant, then the minimum risk is attained at $\eta_{i}=\mu / \sigma^{2}+\sqrt{k / p}$ .

Problem 4.3.7

If $\mathbf{X}$ has the distribution $p_{\theta }(\mathbf{x})$ of (1.5.1) show that, similar to Theorem 3.2, $\mathbb {E}\left[\mathcal{T} \boldsymbol {\eta }(\theta )\right]= \nabla \log m_{\pi }(\mathbf{x})-\nabla \log h(\mathbf{x})$ .

Origin: Stated according to the original source

Problem 4.3.8

(a)(b)(c)

Origin: Stated according to the original source

(a)

Use Stein's identity (Lemma 1.5.15) to show that if $X_{i} \sim p_{\eta_{i}}(x)$ of (3.3.18), then

\mathbb {E}_{\eta }\left[-\nabla \log h(\mathbf{X})\right]=\sum _{i} \eta _{i} \mathbb {E}_{\eta }\left[\frac{\partial }{\partial X_{j}} T_{i}(\mathbf{X})\right].

(b)

If $X_{i}$ are iid from a gamma distribution $\operatorname {Gamma}(a, b)$ , where the shape parameter $a$ is known, use part (a) to find an unbiased estimator of $1 / b$ .

(c)

If the $X_{i}$ are iid from a beta $(a, b)$ distribution, can the identity in part (a) be used to obtain an unbiased estimator of $a$ when $b$ is known, or an unbiased estimator of $b$ when $a$ is known?

Problem 4.3.9

(a)(b)

For the natural exponential family $p_{\eta }(x)$ of (3.3.7) and the conjugate prior $\pi (\eta \mid k, \mu )$ of (3.3.19) establish that:

Origin: Stated according to the original source

(a)

$\mathbb {E}\left[X\right]=A^{\prime }(\eta )$ and var $X=A^{\prime \prime }(\eta )$ , where the expectation is with respect to the sampling density $p_{\eta }(x)$ .

(b)

$\mathbb {E}\left[A^{\prime }(\eta )\right]=\mu$ and $\operatorname {var}[A(\eta )]=(1 / k) \mathbb {E}\left[A^{\prime \prime }(\eta )\right]$ , where the expectation is with respect to the prior distribution.

[The results in part (b) enable us to think of $\mu$ as a prior mean and $k$ as a prior sample size.]

Problem 4.3.10

(a)(b)(c)

For each of the following situations, write the density in the form (3.7), and identify the natural parameter. Obtain the Bayes estimator of $A^{\prime }(\eta )$ using squared loss and the conjugate prior. Express your answer in terms of the original parameters.

Origin: Stated according to the original source

(a)

$X \sim$ binomial $(p, n)$ ,

(b)

$X \sim$ Poisson $(\lambda )$ , and

(c)

$X \sim \operatorname {Gamma}(a, b)$ , $a$ known.

Problem 4.3.11

For the situation of Problem 3.9, if $X_{1}, \ldots , X_{n}$ are iid as $p_{\eta }(x)$ and the prior is the conjugate $\pi (\eta \mid k, \mu )$ , then the posterior distribution is $\pi \left(\eta \mid k+n, \frac{k \mu +n \bar{x}}{k+n}\right)$ .

Origin: Stated according to the original source

Problem 4.3.12

(a)(b)(c)

If $X_{1}, \ldots , X_{n}$ are iid from a one-parameter exponential family, the Bayes estimator of the mean, under squared error loss using a conjugate prior, is of the form $a \bar{X}+b$ for constants $a$ and $b$ .

Origin: Stated according to the original source

(a)

If $\mathbb {E}\left[X_{i}\right]=\mu$ and var $X_{i}=\sigma^{2}$ , then no matter what the distribution of the $X_{i}$ 's, the mean squared error is

E[(a \bar{X}+b)-\mu ]^{2}=a^{2} \text{ var } \bar{X}+[(a-1) \mu +b]^{2}.

(b)

If $\mu$ is unbounded, then no estimator of the form $a \bar{X}+b$ can have finite mean squared error for $a \neq 1$ .

(c)

Can a conjugate-prior Bayes estimator in an exponential family have finite mean squared error?

Remark.

Origin: Stated according to the original source

This problem shows why conjugate-prior Bayes estimators are considered "non-robust."

Section 4.4Equivariant Bayes[14/0%]

Problem 4.4.1

(a)(b)

For the situation of Example 4.2:

Origin: Stated according to the original source

(a)

Show that the Bayes rule under a beta $(\alpha , \alpha )$ prior is equivariant.

(b)

Show that the Bayes rule under any prior that is symmetric about $1 / 2$ is equivariant.

Problem 4.4.2

The Bayes estimator of $\eta$ in Example 4.7 is given by (4.22).

Origin: Stated according to the original source

Problem 4.4.3

The Bayes estimator of $\tau$ in Example 4.5 is given by (4.22).

Origin: Stated according to the original source

Problem 4.4.4

The Bayes estimators of $\eta$ and $\tau$ in Example 4.9 are given by (4.31) and (4.32). (Recall Corollary 1.2.)

Origin: Stated according to the original source

Problem 4.4.5

(a)(b)(c)

For each of the following situations, find a group $G$ that leaves the model invariant and determine left- and right-invariant measures over $G$ . The joint density of $\mathbf{X}= \left(X_{1}, \ldots , X_{n}\right)$ and $\mathbf{Y}=\left(Y_{1}, \ldots , Y_{n}\right)$ and the estimand are

Origin: Stated according to the original source

(a)

$f(\mathbf{x}-\eta , \mathbf{y}-\zeta )$ , estimand $\eta -\zeta$ ;

(b)

$f\left(\frac{\mathbf{x}-\eta }{\sigma }, \frac{\mathbf{y}-\zeta }{\tau }\right)$ , estimand $\tau / \sigma$ ;

(c)

$f\left(\frac{\mathbf{x}-\eta }{\tau }, /, \frac{\mathbf{y}-\zeta }{\tau }\right), \tau$ unknown; estimand $\eta -\zeta$ .

Problem 4.4.6

For each of the situations of Problem 4.5, determine the MRE estimator if the loss is squared error with a scaling that makes it invariant.

Origin: Stated according to the original source

Problem 4.4.7

(a)(b)

For each of the situations of Problem 4.5:

Origin: Stated according to the original source

(a)

Determine the measure over $\Omega$ induced by the right-invariant Haar measure over $\bar{G}$ ;

(b)

Determine the Bayes estimator with respect to the measure found in part (a), and show that it coincides with the MRE estimator.

Problem 4.4.8

In Example 4.9, show that the estimator

\hat{\tau }(\mathbf{x})=\frac{\iint \frac{1}{v^{r}} f\left(\frac{x_{1}-u}{v}, \ldots , \frac{x_{n}-u}{v}\right) d v d u}{\iint \frac{1}{v^{r+1}} f\left(\frac{x_{1}-u}{v}, \ldots , \frac{x_{n}-u}{v}\right) d v d u}

is equivariant under scale changes; that is, it satisfies $\bar{\tau }(c \mathbf{x})=c \hat{\tau }(\mathbf{x})$ for all values of $r$ for which the integrals in $\hat{\tau }(\mathbf{x})$ exist.

Origin: Stated according to the original source

Problem 4.4.9

If $\Lambda$ is a left-invariant measure over $G$ , show that $\Lambda^{*}$ defined by $\Lambda^{*}(B)=\Lambda \left(B^{-1}\right)$ is right invariant, where $B^{-1}=\left\{ g^{-1}: g \in B\right\}$ .

Origin: Stated according to the original source

Problem 4.4.10

(a)(b)(c)

There is a correspondence between Haar measures and Jeffreys priors in the location and scale cases.

Origin: Stated according to the original source

(a)

Show that in the location parameter case, the Jeffreys prior is equal to the invariant Haar measure.

(b)

Show that in the scale parameter case, the Jeffreys prior is equal to the invariant Haar measure.

(c)

Show that in the location-scale case, the Jeffreys prior is equal to the left invariant Haar measure.

[Part c) is a source of some concern because, as mentioned in Section 4.4 (see the discussion following Example 4.9), the best-equivariant rule is Bayes against the rightinvariant Haar measure (if it exists).]

Problem 4.4.11

For the model (3.3.23), find a measure $\nu$ in the $(\xi , \tau )$ plane which remains invariant under the transformations (3.3.24). The next three problems contain a more formal development of left- and right-invariant Haar measures.

Origin: Stated according to the original source

Problem 4.4.12

(a)(b)

A measure $\Lambda$ over a group $G$ is said to be right invariant if it satisfies $\Lambda (B g)=\Lambda (B)$ and left invariant if it satisfies $\Lambda (g B)=\Lambda (B)$ . Note that if $G$ is commutative, the two definitions agree.

Origin: Stated according to the original source

(a)

If the elements $g \in G$ are real numbers $(-\infty <g<\infty )$ and group composition is $g_{2} \cdot g_{1}=g_{1}+g_{2}$ , the measure $\nu$ defined by $\nu (B)=\int_{B} d x$ (i.e., Lebesgue measure) is both left and right invariant.

(b)

If the elements $g \in G$ are the positive real numbers, and composition of $g_{2}$ and $g_{1}$ is multiplication of the two numbers, the measure $\nu$ defined by $\nu (B)=\int_{B}(1 / y) d y$ is both left and right invariant.

Problem 4.4.13

If the elements $g \in G$ are pairs of real numbers $(a, b)$ , $b>0$ , corresponding to the transformations $g x=a+b x$ , group composition by (1.4.8) is

\left(a_{2}, b_{2}\right) \cdot \left(a_{1}, b_{1}\right)=\left(a_{2}+a_{1} b_{2}, b_{1} b_{2}\right).

Of the measures defined by

\nu (B)=\iint _{B} \frac{1}{y} d x d y \quad \text{ and } \quad \nu (B)=\iint _{B} \frac{1}{y^{2}} d x d y

the first is right but not left invariant, and the second is left but not right invariant.

Origin: Stated according to the original source

Problem 4.4.14

The four densities defining the measures $v$ of Problem 4.12 and $4.13(d x,(1 / y) d y$ , $(1 / y) d x d y,\left(1 / y^{2}\right) d x d y$ ) are the only densities (up to multiplicative constants) for which $v$ has the stated invariance properties in the situations of these problems.

Origin: Stated according to the original source

Section 4.5Hierarchical Bayes[21/0%]

Problem 4.5.1

For the model (3.3.1), let $\pi (\theta \mid x, \lambda )$ be a single-prior Bayes posterior and $\pi (\theta \mid x)$ be a hierarchical Bayes posterior. Show that $\pi (\theta \mid x)=\int \pi (\theta \mid x, \lambda ) \cdot \pi (\lambda \mid x) d \lambda$ , where $\pi (\lambda \mid x)=\int f(x \mid \theta ) \pi (\theta \mid \lambda ) \gamma (\lambda ) d \theta / \iint f(x \mid \theta ) \pi (\theta \mid \lambda ) \gamma (\lambda ) d \theta d \lambda$ .

Origin: Stated according to the original source

Problem 4.5.2

(a)(b)

For the situation of Problem 5.1, show that:

Origin: Stated according to the original source

(a)

$\mathbb {E}\left[\theta \mid x\right]=E[\mathbb {E}\left[\theta \mid x, \lambda \right]]$ ;

(b)

$\operatorname {var}(\theta \mid x)=E[\operatorname {var}(\theta \mid x, \lambda )]+\operatorname {var}[\mathbb {E}\left[\theta \mid x, \lambda \right]]$ ;

and hence that $\pi (\theta \mid x)$ will tend to have a larger variance than $\pi \left(\theta \mid x, \lambda_{0}\right)$ .

Problem 4.5.3

(a)(b)

For the model (3.3.3), show that:

Origin: Stated according to the original source

(a)

The marginal prior of $\theta$ , unconditional on $\tau^{2}$ , is given by

\pi (\theta )=\frac{\Gamma \left(a+\frac{1}{2}\right)}{\sqrt{2 \pi } \Gamma (a) b^{a}} \frac{1}{\left(\frac{1}{b}+\frac{\theta ^{2}}{2}\right)^{a+1 / 2}},

which for $a=v / 2$ and $b=2 / v$ is Student's $t$ -distribution with $v$ degrees of freedom.

(b)

The marginal posterior of $\tau^{2}$ is given by

\pi \left(\tau ^{2} \mid \bar{x}\right)=\frac{\left[\frac{\sigma ^{2} \tau ^{2}}{\sigma ^{2}+\tau ^{2}}\right]^{1 / 2} e^{-\frac{1}{2} \frac{\bar{x}^{2}}{\sigma ^{2}+\tau ^{2}}} \frac{1}{\left(\tau ^{2}\right)^{a+3 / 2}} e^{-1 / b \tau ^{2}}}{\int _{0}^{\infty }\left[\frac{\sigma ^{2} \tau ^{2}}{\sigma ^{2}+\tau ^{2}}\right]^{1 / 2} e^{-\frac{1}{2} \frac{\bar{x}^{2}}{\sigma ^{2}+\tau ^{2}}} \frac{1}{\left(\tau ^{2}\right)^{a+3 / 2}} e^{-1 / b \tau ^{2}} d \tau ^{2}}

Problem 4.5.4

(a)(b)(c)

Albert and Gupta (1985) investigate theory and applications of the hierarchical model

\begin{align} X_{i} \mid \theta _{i} & \sim b\left(\theta _{i}, n\right), \quad i=1, \ldots , p, \text{ independent, } \\ \theta _{i} \mid \eta & \sim \operatorname {beta}[k \eta , k(1-\eta )], k \text{ known, } \\ \eta & \sim \operatorname {Uniform}(0,1). \end{align}

Origin: Stated according to the original source

(a)

Show that

\begin{align} \mathbb {E}\left[\theta _{i} \mid \mathbf{x}\right] & =\left(\frac{n}{n+k}\right)\left(\frac{x_{i}}{n}\right)+\left(\frac{k}{n+k}\right) \mathbb {E}\left[\eta \mid \mathbf{x}\right], \\ \operatorname {var}\left(\theta _{i} \mid \mathbf{x}\right) & =\frac{k^{2}}{(n+k)(n+k+1)} \operatorname {var}(\eta \mid \mathbf{x}). \end{align}

[Note that $\mathbb {E}\left[\eta \mid \mathbf{x}\right]$ and $\operatorname {var}(\eta \mid \mathbf{x})$ are not expressible in a simple form.]

(b)

Unconditionally on $\eta$ , the $\theta_{i}$ 's have conditional covariance

\operatorname {cov}\left(\theta _{i}, \theta _{j} \mid \mathbf{x}\right)=\left(\frac{k}{n+k}\right)^{2} \operatorname {var}(\eta \mid \mathbf{x}), \quad i \neq j.

(c)

Ignoring the prior distribution of $\eta$ , show how to construct an empirical Bayes estimator of $\theta_{i}$ . (Again, this is not expressible in a simple form.)

[Albert and Gupta (1985) actually consider a more general model than given here, and show how to approximate the Bayes solution. They apply their model to a problem of nonresponse in mail surveys.]

Problem 4.5.5

(a)(b)(c)

Origin: Stated according to the original source

(a)

Analogous to Problem 1.7.9, establish that for any random variable $X, Y$ , and $Z$ ,

\operatorname {cov}(X, Y)=E[\operatorname {cov}(X, Y) \mid Z]+\operatorname {cov}[\mathbb {E}\left[X \mid Z\right], \mathbb {E}\left[Y \mid Z\right]].

(b)

For the hierarchy

\begin{align} X_{i} \mid \theta _{i} & \sim f\left(x \mid \theta _{i}\right), \quad i=1, \ldots , p, \text{ independent }, \\ \Theta _{i} \mid \lambda & \sim \pi \left(\theta _{i} \mid \lambda \right), \quad i=1, \ldots , p, \text{ independent }, \\ \Lambda & \sim \gamma (\lambda ), \end{align}

show that $\operatorname {cov}\left(\Theta_{i}, \Theta_{j} \mid \mathbf{x}\right)=\operatorname {cov}\left[\mathbb {E}\left[\Theta_{i} \mid \mathbf{x}, \lambda \right], \mathbb {E}\left[\Theta_{j} \mid \mathbf{x}, \lambda \right]\right]$ .

(c)

If $\mathbb {E}\left[\Theta_{i} \mid \mathbf{x}, \lambda \right]=g\left(x_{i}\right)+h(\lambda ), i=1, \ldots , p$ , where $g(\cdot )$ and $h(\cdot )$ are known, then

\operatorname {cov}\left(\Theta _{i}, \Theta _{j} \mid \mathbf{x}\right)=\operatorname {var}\left[\mathbb {E}\left[\Theta _{i} \mid \mathbf{x}, \lambda \right]\right].

[Part (c) points to what can be considered a limitation in the applicability of some hierarchical models, that they imply a positive correlation structure in the posterior distribution.]

Problem 4.5.6

The one-way random effects model of Example 2.7 (see also Examples 3.5.1 and 3.5.5) can be written as the hierarchical model

\begin{align} X_{i j} \mid \mu , \alpha _{i} & \sim N\left(\mu +\alpha _{i}, \sigma ^{2}\right), \quad j=1, \ldots , n, \quad i=1, \ldots , s, \\ \alpha _{i} & \sim N\left(0, \sigma _{A}^{2}\right), \quad i=1, \ldots , s. \end{align}

If, in addition, we specify that $\mu \sim$ Uniform $(-\infty , \infty )$ , show that the Bayes estimator of $\mu +\alpha_{i}$ under squared error loss is given by (3.5.13), the UMVU predictor of $\mu +\alpha_{i}$ .

Origin: Stated according to the original source

Problem 4.5.7

(a)(b)

Referring to Example 6.6:

Origin: Stated according to the original source

(a)

Using the prior distribution for $\gamma (b)$ given in (5.6.27), show that the mode of the posterior distribution $\pi (b \mid \mathbf{x})$ is $\hat{b}=(p \bar{x}+\alpha -1) /(p a+\beta -1)$ , and hence the empirical Bayes estimator based on this $\hat{b}$ does not equal the hierarchical Bayes estimator (5.6.29).

(b)

Show that if we estimate $b /(b+1)$ using its posterior expectation $E[b /(b+1) \mid \mathbf{x}]$ , then the resulting empirical Bayes estimator is equal to the hierarchical Bayes estimator.

Problem 4.5.8

(a)(b)(c)(d)

The method of Monte Carlo integration allows the calculation of (possibly complicated) integrals by using (possibly simple) generations of random variables.

Origin: Stated according to the original source

(a)

To calculate $\int h(x) f_{X}(x) d x$ , generate a sample $X_{1}, \ldots , X_{m}$ , iid, from $f_{X}(x)$ . Then, $1 / m \sum_{i=1}^{m} h\left(x_{i}\right) \rightarrow \int h(x) f_{X}(x) d x$ as $m \rightarrow \infty$ .

(b)

If it is difficult to generate random variable from $f_{X}(x)$ , then generate pairs of random variables

\begin{align} Y_{i} & \sim f_{Y}(y), \\ X_{i} & \sim f_{X \mid Y}\left(x \mid y_{i}\right). \end{align}

Then, $1 / m \sum_{i=1}^{m} h\left(x_{i}\right) \rightarrow \int h(x) f_{X}(x) d x$ as $m \rightarrow \infty$ . [Show that if $X$ is generated according to $Y \sim f_{Y}(y)$ and $X \sim f_{X \mid Y}(x \mid Y)$ , then $\mathbb {P}\left(X \leq a\right)=\int_{-\infty }^{a} f_{x}(x) d x$ .]

(c)

If it is difficult to generate as in part (b), then generate

\begin{align} X_{m_{i}} & \sim f_{X \mid Y}\left(x \mid Y_{m_{i-1}}\right), \\ Y_{m_{i}} & \sim f_{Y \mid X}\left(y \mid X_{m_{i}}\right). \end{align}

for $i=1, \ldots , K$ and $m=1, \ldots , M$ .

(d)

Show that:

for each $m,\left\{ X_{m_{i}}\right\}$ is a Markov chain. If it is also an ergodic Markov chain $X_{m_{i}} \xrightarrow {\mathcal{L}} X$ , as $i \rightarrow \infty$ , where $X$ has the stationary distribution of the chain.
If the stationary distribution of the chain is $f_{X}(x)$ , then
$\frac{1}{M} \sum _{m=1}^{M} h\left(x_{m_{k}}\right) \rightarrow \int h(x) f_{X}(x) d x$
as $K, M \rightarrow \infty$ .

Remark.

Origin: Stated according to the original source

This is the basic theory behind the Gibbs sampler. For each $k$ , we have generated independent random variables $X_{m_{k}}, m=1, \ldots , M$ , where $X_{m_{k}}$ is distributed according to $f_{X \mid Y}\left(x \mid y_{m_{k-}}\right)$ . It is also the case that for each $m$ and large $k, X_{m_{k}}$ is approximately distributed according to $f_{X}(x)$ , although the variables are not now independent. The advantages and disadvantages of these computational schemes (one-long-chain vs. manyshort-chains) are debated in Gelman and Rubin 1992; see also Geyer and Thompson 1992 and Smith and Roberts 1992. The prevailing consensus leans toward one long chain.

Problem 4.5.9

(a)(b)(c)

To understand the convergence of the Gibbs sampler, let $(X, Y) \sim f(x, y)$ , and define

k\left(x, x^{\prime }\right)=\int f_{X \mid Y}(x \mid y) f_{Y \mid X}\left(y \mid x^{\prime }\right) d y

Origin: Stated according to the original source

(a)

Show that the function $h^{*}(\cdot )$ that solves $h^{*}(x)=\int k\left(x, x^{\prime }\right) h^{*}\left(x^{\prime }\right) d x^{\prime }$ is $h^{*}(x)= f_{X}(x)$ , the marginal distribution of $X$ .

(b)

Write down the analogous integral equation that is solved by $f_{Y}(y)$ .

(c)

Define a sequence of functions recursively by $h_{i+1}(x)=\int k\left(x, x^{\prime }\right) h_{i}\left(x^{\prime }\right) d x_{I}^{\prime }$ where $h_{0}(x)$ is arbitrary but satisfies $\sup_{x}\left|\frac{h_{0}(x)}{h^{*}(x)}\right|<\infty$ . Show that

\int \left|h_{i+1}(x)-h^{*}(x)\right| d x<\int \left|h_{i}(x)-h^{*}(x)\right| d x

and, hence, $h_{i}(x)$ converges to $h^{*}(x)$ .

[The method of part (c) is called successive substitution. When there are two variables in the Gibbs sampler, it is equivalent to data augmentation (Tanner and Wong 1987). Even if the variables are vector-valued, the above results establish convergence. If the original vector of variables contains more than two variables, then a more general version of this argument is needed (Gelfand and Smith 1990).]

Problem 4.5.10

(a)(b)

A direct Monte Carlo implementation of substitution sampling is provided by the data augmentation algorithm (Tanner and Wong 1987). If we define

h_{i+1}(x)=\int \left[\int f_{X \mid Y}(x \mid y) f_{Y \mid X}\left(y \mid x^{\prime }\right) d y\right] h_{i}\left(x^{\prime }\right) d x^{\prime }

then from Problem 5.9, $h_{i}(x) \rightarrow f_{x}(x)$ as $i \rightarrow \infty$ .

Origin: Stated according to the original source

(a)

To calculate $h_{i+1}$ using Monte Carlo integration:

Generate $X_{j}^{\prime } \sim h_{i}\left(x^{\prime }\right), \quad j=1, \ldots , J$ .
Generate, for each $x_{j}^{\prime }, Y_{j k} \sim f_{Y \mid X}\left(y \mid x_{j}^{\prime }\right), k=1, \ldots , K$ .
Calculate $\hat{h}_{i+1}(x)=\frac{1}{J} \sum_{j=1}^{J} \frac{1}{K} \sum_{k=1}^{K} f_{X \mid Y}\left(x \mid y_{j k}\right)$ . Then, $\hat{h}_{i+1}(x) \rightarrow h_{i+1}(x)$ as $J, K \rightarrow \infty$ , and hence the data augmentation algorithm converges.

(b)

To implement (a)(i), we must be able to generate a random variable from a mixture distribution. Show that if $f_{Y}(y)=\sum_{i=1}^{n} a_{i} g_{i}(y), \Sigma a_{i}=1$ , then the algorithm

Select $g_{i}$ with probability $a_{i}$
Generate $Y \sim g_{i}$ produces a random variable with distribution $f_{Y}$ .

Hence, show how to implement step (a)(i) by generating random variables from $f_{X \mid Y}$ . Tanner and Wong (1987) note that this algorithm will work even if $J=1$ , which yields the approximation

\hat{h}_{i+1}(x)=\frac{1}{K} \sum _{k=1}^{K} f_{X \mid Y}\left(x \mid y_{k}\right),

identical to the Gibbs sampler. The data augmentation algorithm can also be seen as an application of the process of multiple imputation (Rubin 1976, 1987, Little and Rubin 1987).

Problem 4.5.11

(a)(b)(c)

Successive substitution sampling can be implemented via the Gibbs sampler in the following way. From Problem 5.8(c), we want to calculate

h_{M}=\frac{1}{M} \sum _{m=1}^{M} k\left(x \mid x_{m_{k}}\right)=\frac{1}{M} \sum _{m=1}^{M} \int f_{X \mid Y}(x \mid y) f_{X \mid Y}\left(y \mid x_{m_{k}}\right) d y.

Origin: Stated according to the original source

(a)

Show that $h_{M}(x) \rightarrow f_{X}(x)$ as $M \rightarrow \infty$ .

(b)

Given $x_{m_{k}}$ , a Monte Carlo approximation to $h_{M}(x)$ is

\hat{h}_{M}(x)=\frac{1}{M} \sum _{m=1}^{M} \frac{1}{J} \sum _{j=1}^{J} f_{X \mid Y}\left(x \mid y_{k_{j}}\right)

where $Y_{k_{j}} \sim f_{Y \mid X}\left(y \mid x_{m_{k}}\right)$ and $\hat{h}_{M}(x) \rightarrow h_{M}(x)$ as $J \rightarrow \infty$ .

(c)

Hence, as $M, J \rightarrow \infty , \hat{h}_{M}(x) \rightarrow f_{X}(x)$ .

Remark.

Origin: Stated according to the original source

This is the Gibbs sampler, which is usually implemented with $J=1$ .

Problem 4.5.12

(a)(b)(c)

For the situation of Example 5.6, show that

Origin: Stated according to the original source

(a)

\mathbb {E}\left[\frac{1}{M} \sum _{i=1}^{M} \Theta _{i}\right]=\mathbb {E}\left[\frac{1}{M} \sum _{i=1}^{M} \mathbb {E}\left[\Theta \mid \mathbf{x}, \tau _{i}\right]\right],

(b)

\operatorname {var}\left(\frac{1}{M} \sum _{i=1}^{M} \Theta _{i}\right) \geq \operatorname {var}\left(\frac{1}{M} \sum _{i=1}^{M} \mathbb {E}\left[\Theta \mid \mathbf{x}, \tau _{i}\right]\right).

(c)

Discuss when equality might hold in (b). Can you give an example?

Problem 4.5.13

Show that for the hierarchy (5.5.1), the posterior distributions $\pi (\theta \mid \mathbf{x})$ and $\pi (\lambda \mid \mathbf{x})$ satisfy

\begin{align} & \pi (\theta \mid \mathbf{x})=\int \left[\int \pi (\theta \mid \mathbf{x}, \lambda ) \pi \left(\lambda \mid \mathbf{x}, \theta ^{\prime }\right) d \lambda \right] \pi \left(\theta ^{\prime } \mid \mathbf{x}\right) d \theta ^{\prime } \\ & \pi (\lambda \mid \mathbf{x})=\int \left[\int \pi (\lambda \mid \mathbf{x}, \theta ) \pi \left(\theta \mid \mathbf{x}, \lambda ^{\prime }\right) d \theta \right] \pi \left(\lambda ^{\prime } \mid \mathbf{x}\right) d \lambda ^{\prime } \end{align}

and, hence, are stationary points of the Markov chains in (5.5.13).

Origin: Stated according to the original source

Problem 4.5.14

(a)(b)(c)(d)

Starting from a uniform random variable $U \sim \operatorname {Uniform}(0,1)$ , it is possible to construct many random variables through transformations.

Origin: Stated according to the original source

(a)

Show that $-\log U \sim \exp (1)$ .

(b)

Show that $-\sum_{i=1}^{n} \log U_{i} \sim \operatorname {Gamma}(n, 1)$ , where $U_{1}, \ldots , U_{n}$ are iid as $U(0,1)$ .

(c)

Let $X \sim \operatorname {Exp}(a, b)$ . Write $X$ as a function of $U$ .

(d)

Let $X \sim \operatorname {Gamma}(n, \beta )$ , $n$ an integer. Write $X$ as a function of $U_{1}, \ldots , U_{n}$ , iid as $U(0,1)$ .

Problem 4.5.15

(a)(b)(c)

Starting with a $U(0,1)$ random variable, the transformations of Problem 5.14 will not get us normal random variables, or gamma random variables with noninteger shape parameters. One way of doing this is to use the Accept-Reject Algorithm (Ripley 1987, Section 3.2), an algorithm for simulating $X \sim f(x)$ :

Generate $Y \sim g(y), U \sim U(0,1)$ , independent.
Calculate $\rho (Y)=\frac{1}{M} \frac{f(Y)}{g(Y)}$ where $M=\sup_{t} f(t) / g(t)$ .
If $U<\rho (Y)$ , set $X=Y$ , otherwise return to i).

Origin: Stated according to the original source

(a)

Show that the algorithm will generate $X \sim f(x)$ .

(b)

Starting with $Y \sim \exp (1)$ , show how to generate $X \sim N(0,1)$ .

(c)

Show how to generate a gamma random variable with a noninteger shape parameter.

Problem 4.5.16

(a)(b)

Consider the normal hierarchical model

\begin{align} X \mid \theta _{1} & \sim n\left(\theta _{1}, \sigma _{1}^{2}\right), \\ \theta _{1} \mid \theta _{2} & \sim n\left(\theta _{2} \mid \sigma _{2}^{2}\right), \\ \vdots & \\ \theta _{k-1} \mid \theta _{k} & \sim n\left(\theta _{k}, \sigma _{k}^{2}\right) \end{align}

where $\sigma_{1}^{2}, i=1, \ldots , k$ , are known.

Origin: Stated according to the original source

(a)

Show that the posterior distribution of $\theta_{i}(1 \leq i \leq k-1)$ is

\pi \left(\theta _{i} \mid x, \theta _{k}\right)=N\left(\alpha _{i} x+\left(1-\alpha _{i}\right) \theta _{k}, \tau _{i}^{2}\right)

where $\tau_{i}^{2}=\left(\Sigma_{1}^{i} \sigma_{j}^{2}\right)\left(\Sigma_{i+1}^{k} \sigma_{j}^{2}\right) / \Sigma_{i}^{k} \sigma_{j}^{2}$ and $\alpha_{i}=\tau_{i}^{2} / \Sigma_{1}^{i} \sigma_{j}^{2}$ .

(b)

Find an expression for the Kullback-Leibler information $K\left[\pi \left(\theta_{i} \mid x, \theta_{k}\right), \pi \left(\theta_{i} \mid \theta_{k}\right)\right]$ and show that it is a decreasing function of $i$ .

Problem 4.5.17

(a)(b)(c)

The original proof of Theorem 5.7 (Goel and DeGroot 1981) used Rényi's entropy function (Rényi 1961)

R_{\alpha }(f, g)=\frac{1}{\alpha -1} \log \int f^{\alpha }(x) g^{1-\alpha }(x) d \mu (x),

where $f$ and $g$ are densities, $\mu$ is a dominating measure, and $\alpha$ is a constant, $\alpha \neq 1$ .

Origin: Stated according to the original source

(a)

Show that $R_{\alpha }(f, g)$ satisfies $R_{\alpha }(f, g)>0$ and $R_{\alpha }(f, f)=0$ .

(b)

Show that Theorem 5.7 holds if $R_{\alpha }(f, g)$ is used instead of $K[f, g]$ .

(c)

Show that $\lim_{\alpha \rightarrow 1} R_{\alpha }(f, g)=K[f, g]$ , and provide another proof of Theorem 5.7.

Problem 4.5.18

The Kullback-Leibler information, $K[f, g]$ (5.5.25), is not symmetric in $f$ and $g$ , and a modification, called the divergence, remedies this. Define $J[f, g]$ , the divergence between $f$ and $g$ , to be $J[f, g]=K[f, g]+K[g, f]$ . Show that, analogous to Theorem 5.7, $J[\pi (\lambda \mid x), \gamma (\lambda )]<J[\pi (\theta \mid x), \pi (\theta )]$ .

Origin: Stated according to the original source

Problem 4.5.19

Goel and DeGroot (1981) define a Bayesian analog of Fisher information [see (2.5.10)] as

\mathcal{I}[\pi (\theta \mid x)]=\int _{\Omega }\left[\frac{\frac{\partial }{\partial x} \pi (\theta \mid x)}{\pi (\theta \mid x)}\right]^{2} d \theta

the information that $x$ has about the posterior distribution. As in Theorem 5.7, show that $\mathcal{I}[\pi (\lambda \mid x)]<\mathcal{I}[\pi (\theta \mid x)]$ , again showing that the influence of $\lambda$ is less than that of $\theta$ .

Origin: Stated according to the original source

Problem 4.5.20

(a)(b)(c)

Each of $m$ spores has a probability $\tau$ of germinating. Of the $r$ spores that germinate, each has probability $\omega$ of bending in a particular direction. If $s$ bends in the particular direction, a probability model to describe this process is the bivariate binomial, with mass function

f(r, s \mid \tau , \omega , m)=\binom {m}{r} \tau ^{r}(1-\tau )^{m-r}\binom {r}{s} \omega ^{s}(1-\omega )^{r-s}.

Origin: Stated according to the original source

(a)

Show that the Jeffreys prior is $\pi_{J}(\tau , \omega )=(1-\tau )^{-1 / 2} \omega^{-1 / 2}(1-\omega )^{-1 / 2}$ .

(b)

If $\tau$ is considered a nuisance parameter, the reference prior is

\pi _{R}(\tau , \omega )=\tau ^{-1 / 2}(1-\tau )^{-1 / 2} \omega ^{-1 / 2}(1-\omega )^{-1 / 2}.

Compare the posterior means $\mathbb {E}\left[\omega \mid r, s, m\right]$ under both the Jeffreys and reference priors. Is one more appropriate?

(c)

What is the effect of the different priors on the posterior variance?

[Priors for the bivariate binomial have been considered by Crowder and Sweeting (1989), Polson and Wasserman (1990), and Clark and Wasserman (1993), who propose a reference/Jeffreys trade-off prior.]

Problem 4.5.21

(a)(b)

Let $\mathcal{F}=\left\{ f(x \mid \theta ) ; \theta \in \Omega \right\}$ be a family of probability densities. The Kullback-Leibler information for discrimination between two densities in $\mathcal{F}$ can be written

\psi \left(\theta _{1}, \theta _{2}\right)=\int f\left(x \mid \theta _{1}\right) \log \left[\frac{f\left(x \mid \theta _{1}\right)}{f\left(x \mid \theta _{2}\right)}\right] d x.

Recall that the gradient of $\psi$ is $\nabla \psi =\left\{ \left(\partial / \partial \theta_{i}\right) \psi \right\}$ and the Hessian is $\nabla \nabla \psi = \left\{ \left(\partial^{2} / \partial \theta_{i} \partial \theta_{j}\right) \psi \right\}$ .

Origin: Stated according to the original source

(a)

If integration and differentiation can be interchanged, show that

\nabla \psi (\theta , \theta )=0 \quad \text{ and } \quad \operatorname {det}[\nabla \nabla \psi (\theta , \theta )]=I(\theta ),

where $I(\theta )$ is the Fisher information of $f(x \mid \theta )$ .

(b)

George and McCulloch (1993) argue that choosing $\pi (\theta )=(\operatorname {det}[\nabla \nabla \psi (\theta , \theta )])^{1 / 2}$ is an appealing least informative choice of priors. What justification can you give for this?

Section 4.6Empirical Bayes[21/0%]

Problem 4.6.1

For the model (3.3.1), show that $\left.\delta^{\lambda }(x)\right|_{\lambda =\hat{\lambda }}=\delta^{\hat{\lambda }(x)}$ , where the Bayes estimator $\delta^{\lambda }(x)$ minimizes $\int L[\theta , d(x)] \pi (\theta \mid x, \lambda ) d \theta$ and the empirical Bayes estimator $\delta^{\hat{\lambda }(x)}$ minimizes $\int L[\theta , d(x)] \pi (\theta \mid x, \hat{\lambda }) d \theta$ .

Origin: Stated according to the original source

Problem 4.6.2

(a)(b)

This problem will investigate conditions under which an empirical Bayes estimator is a Bayes estimator. Expression (6.6.3) is a true posterior expected loss if $\pi (\theta \mid \mathbf{x}, \hat{\lambda }(\mathbf{x}))$ is a true posterior.

From the hierarchy

\begin{align} & X \mid \theta \sim f(x \mid \theta ), \\ & \Theta \mid \lambda \sim \pi (\theta \mid \lambda ), \end{align}

define the joint distribution of $\mathbf{X}$ and $\Theta$ to be $(\mathbf{X}, \theta ) \sim g(\mathbf{x}, \theta )=f(\mathbf{x} \mid \theta ) \pi (\theta \mid \hat{\lambda }(\mathbf{x}))$ , where $\pi (\theta \mid \hat{\lambda }(\mathbf{x}))$ is obtained by substituting $\hat{\lambda }(\mathbf{x})$ for $\lambda$ in $\pi (\theta \mid \lambda )$ .

Origin: Stated according to the original source

(a)

Show that, for this joint density, the formal Bayes estimator is equivalent to the empirical Bayes estimator from the hierarchical model.

(b)

If $f(\cdot \mid \theta )$ and $\pi (\cdot \mid \lambda )$ are proper densities, then $\int g(\mathbf{x}, \theta ) d \theta <\infty$ . However, $\iint g(\mathbf{x}, \theta ) d \mathbf{x} d \theta$ need not be finite.

Problem 4.6.3

For the model (6.3.1), the Bayes estimator $\delta^{\lambda }(x)$ minimizes $\int L(\theta , d(x)) \times \pi (\theta \mid x, \lambda ) d \theta$ and the empirical Bayes estimator, $\delta^{\hat{\lambda }}(x)$ , minimizes $\int L(\theta , d(x)) \pi (\theta \mid x, \hat{\lambda }(x)) d \theta$ . Show that $\left.\delta^{\lambda }(x)\right|_{\lambda =\hat{\lambda }(x)}=\delta^{\hat{\lambda }}(x)$ .

Origin: Stated according to the original source

Problem 4.6.4

(a)(b)

For the situation of Example 6.1:

Origin: Stated according to the original source

(a)

Show that

\int _{-\infty }^{\infty } e^{-n / 2 \sigma ^{2}(\bar{x}-\theta )^{2}} e^{(-1 / 2) \theta ^{2} / \tau ^{2}} d \theta =\sqrt{2 \pi }\left(\frac{\sigma ^{2} \tau ^{2}}{\sigma ^{2}+\tau ^{2}}\right)^{1 / 2} e^{(-n / 2) \bar{x}^{2} / \sigma ^{2}+n \tau ^{2}}

and, hence, establish (6.6.4).

(b)

Verify that the marginal MLE of $\sigma^{2}+n \tau^{2}$ is $n \bar{x}^{2}$ and that the empirical Bayes estimator is given by (6.6.5).

Problem 4.6.5

(a)(b)

Referring to Example 6.2:

Origin: Stated according to the original source

(a)

Show that the Bayes risk, $r\left(\pi , \delta^{\pi }\right)$ , of the Bayes estimator (6.6.7) is given by

r\left(\pi , \delta ^{\pi }\right)=k \mathbb {E}\left[\operatorname {var}\left(p_{k} \mid x_{k}\right)\right]=\frac{k a b}{(a+b)(a+b+1)(a+b+n)}.

(b)

Show that the Bayes risk of the unbiased estimator $\mathbf{X} / n=\left(X_{1} / n, \ldots , X_{k} / n\right)$ is given by

r(\pi , \mathbf{X} / n)=\frac{k a b}{n(a+b+1)(a+b)}.

Problem 4.6.6

Theorem 6.3 states: for the situation of Corollary 3.3 (if $X=(X_{1}, \ldots , X_{p})$ has the density $p_{\eta }(x)=e^{\sum_{i} \eta_{i} x_{i}-A(\eta )} h(x)$ and $\eta$ has prior density $\pi (\eta )$ , the Bayes estimator of $\eta$ under squared error loss is $\mathbb {E}\left[\eta_{i} \mid x\right]=\frac{\partial }{\partial x_{i}} \log m(x)-\frac{\partial }{\partial x_{i}} \log h(x)$ ), with prior distribution $\pi (\eta \mid \lambda )$ , suppose $\hat{\lambda }(x)$ is the MLE of $\lambda$ based on $m(x \mid \lambda )$ . Then, the empirical Bayes estimator is

\mathbb {E}\left[\eta _{i} \mid x, \hat{\lambda }\right]=\frac{\partial }{\partial x_{i}} \log m(x \mid \hat{\lambda }(x))-\frac{\partial }{\partial x_{i}} \log h(x).

Extend Theorem 6.3 to the case of Theorem 3.2; that is, if $\mathbf{X}$ has density (3.3.7) and $\boldsymbol {\eta }$ has prior density $\pi (\boldsymbol {\eta } \mid \gamma )$ , then the empirical Bayes estimator is

\mathbb {E}\left[\left.\sum \eta _{i} \frac{\partial T_{i}(\mathbf{x})}{\partial x_{j}} \right\rvert \, \mathbf{x}, \hat{\gamma }(\mathbf{x})\right]=\frac{\partial }{\partial x_{j}} \log m(\mathbf{x} \mid \hat{\gamma }(\mathbf{x}))-\frac{\partial }{\partial x_{j}} \log h(\mathbf{x}),

where $m(\mathbf{x} \mid \gamma )$ is the marginal distribution of X and $\hat{\gamma }(\mathbf{x})$ is the marginal MLE of $\gamma$ .

Origin: Stated according to the original source

Problem 4.6.7

(a)(b)

Origin: Stated according to the original source

(a)

For $p_{\eta }(\mathbf{x})$ of (1.5.2), show that for any prior distribution $\pi (\eta \mid \lambda )$ that is dependent on a hyperparameter $\lambda$ , the empirical Bayes estimator is given by

\mathbb {E}\left[\left.\sum _{i=1}^{s} \eta _{i} \frac{\partial }{\partial x_{j}} T_{i}(\mathbf{x}) \right\rvert \, \mathbf{x}, \hat{\lambda }\right]=\frac{\partial }{\partial x_{j}} \log m_{\pi }(\mathbf{x} \mid \hat{\lambda }(\mathbf{x}))-\frac{\partial }{\partial x_{j}} \log h(\mathbf{x}).

where $m_{\pi }(\mathbf{x})=\int p_{\theta }(\mathbf{x}) \pi (\theta ) d \theta$ .

(b)

If X has the distribution $p_{\theta }(\mathbf{x})$ of (1.5.1), show that a similar formulas holds, that is,

\mathbb {E}\left[\mathcal{T} \eta (\theta ) \mid \hat{\lambda }\right]=\nabla \log m_{\pi }(\mathbf{x} \mid \hat{\lambda })-\nabla \log h(\mathbf{x}),

where $\mathcal{T}=\left\{ \partial T_{i} / \partial x_{j}\right\}$ is the Jacobian of $\mathcal{T}$ and $\nabla a$ is the gradient vector of $a$ , that is, $\nabla a=\left\{ \partial a / \partial x_{i}\right\}$ .

Problem 4.6.8

(a)(b)

For each of the following situations, write the empirical Bayes estimator of the natural parameter (under squared error loss) in the form (6.6.12), using the marginal likelihood estimator of the hyperparameter $\lambda$ . Evaluate the expressions as far as possible.

Origin: Stated according to the original source

(a)

$X_{i} \sim N\left(0, \sigma_{i}^{2}\right), i=1, \ldots , p$ , independent; $1 / \sigma_{i}^{2} \sim \operatorname {Exponential}(\lambda )$ .

(b)

$X_{i} \sim N\left(\theta_{i}, 1\right), \quad i=1, \ldots , p$ , independent, $\theta_{i} \sim D E(0, \lambda )$ .

Problem 4.6.9

(a)(b)

Strawderman (1992) shows that the James-Stein estimator can be viewed as an empirical Bayes estimator in an arbitrary location family. Let $\mathbf{X}_{p \times 1} \sim f(x-\theta )$ , with $\mathbb {E}\left[\mathbf{X}\right]=\theta$ and var $\mathbf{X}=\sigma^{2} I$ . Let the prior be $\theta \sim f^{* n}$ , the $n$ -fold convolution of $f$ with itself. [The convolution of $f$ with itself is $f^{* 2}(x)=\int f(x-y) f(y) d y$ . The $n$ -fold convolution is $f^{* n}(x)=\int f *(n-1)(x)(x-y) f(y) d y$ .] Equivalently, let $U_{i} \sim f, i=0, \cdots , n$ , iid, $\theta =\sum_{1}^{n} U_{i}$ , and $\mathbf{X}=U_{0}+\theta$ .

Origin: Stated according to the original source

(a)

Show that the Bayes rule against squared error loss is $\frac{n}{n+1} \mathbf{x}$ . Note that $n$ is a prior parameter.

(b)

Show that $|\mathbf{X}|^{2} /\left(p \sigma^{2}\right)$ is an unbiased estimator of $n+1$ , and hence that an empirical Bayes estimator of $\theta$ is given by $\delta^{E B}=\left[1-\left(p \sigma^{2} /|\mathbf{x}|^{2}\right)\right] \mathbf{x}$ .

Problem 4.6.10

(a)(b)(c)

Show for the hierarchy of Example 3.4, where $\sigma^{2}$ and $\tau^{2}$ are known but $\mu$ is unknown, that:

Origin: Stated according to the original source

(a)

The empirical Bayes estimator of $\theta_{i}$ , based on the marginal MLE of $\theta_{i}$ , is $\frac{\tau^{2}}{\sigma^{2}+\tau^{2}} X_{i}+ \frac{\sigma^{2}}{\sigma^{2}+\tau^{2}} \bar{X}$ .

(b)

The Bayes risk, under sum-of-squared-errors loss, of the empirical Bayes estimator from part (a) is

p \sigma ^{2}-\frac{2(p-1)^{2} \sigma ^{4}}{p\left(\sigma ^{2}+\tau ^{2}\right)}+(p-1)\left(\frac{\sigma ^{2}}{\sigma ^{2}+\tau ^{2}}\right)^{2} \sum _{i=1}^{p} \mathbb {E}\left[(X_{i}-\bar{X})^{2}\right].

(c)

The minimum risk of the empirical Bayes estimator is attained when all $\theta_{i} \mathrm{~ s}$ are equal. [Hint: Show that $\sum_{i=1}^{p} \mathbb {E}\left[\left(X_{i}-\bar{X}\right)^{2}\right]=\sum_{i=1}^{p}\left(\theta_{i}-\bar{\theta }\right)^{2}+(p-1) \sigma^{2}$ .]

Problem 4.6.11

For $\mathbb {E}\left[\Theta \mid \mathbf{x}\right]$ of (5.5.8), show that as $v \rightarrow \infty , \mathbb {E}\left[\Theta \mid x\right] \rightarrow \left[p /\left(p+\sigma^{2}\right)\right] \bar{x}$ , the Bayes estimator under a $N(0,1)$ prior.

Origin: Stated according to the original source

Problem 4.6.12

(a)(b)(c)

Origin: Stated according to the original source

(a)

Show that the empirical Bayes $\delta^{E B}(\bar{x})=\left(1-\sigma^{2} / \max \left\{ \sigma^{2}, p \bar{x}^{2}\right\} \right) \bar{x}$ of (6.6.5) has bounded mean squared error.

(b)

Show that a variation of $\delta^{E B}(\bar{x})$ , of part (a), $\delta^{v}(\bar{x})=\left[1-\sigma^{2} /\left(v+p \bar{x}^{2}\right)\right] \bar{x}$ , also has bounded mean squared error.

(c)

For $\sigma^{2}=\tau^{2}=1$ , plot the risk functions of the estimators of parts (a) and (b).

[Thompson (1968a, 1968b) investigated the mean squared error properties of estimators like those in part (b). Although such estimators have smaller mean squared error than $\bar{x}$ for small values of $\theta$ , they always have larger mean squared error for larger values of $\theta$ .]

Problem 4.6.13

(a)(b)

Origin: Stated according to the original source

(a)

For the hierarchy (5.5.7), with $\sigma^{2}=1$ and $p=10$ , evaluate the Bayes risk $r\left(\pi , \delta^{\pi }\right)$ of the Bayes estimator (5.5.8) for $v=2,5$ , and 10.

(b)

Calculate the Bayes risk of the estimator $\delta^{v}$ of Problem 6.12(b). Find a value of $v$ that yields a good approximation to the risk of the hierarchical Bayes estimator. Compare it to the Bayes risk of the empirical Bayes estimator of Problem 6.12(a).

Problem 4.6.14

Referring to Example 6.6, show that the empirical Bayes estimator is also a hierarchical Bayes estimator using the prior $\gamma (b)=1 / b$ .

Origin: Stated according to the original source

Problem 4.6.15

(a)(b)(c)

The Taylor series approximation to the estimator (5.5.8) is carried out in a number of steps. Show that:

Origin: Stated according to the original source

(a)

Using a first-order Taylor expansion around the point $\bar{x}$ , we have

\begin{align} \frac{1}{\left(1+\theta ^{2} / v\right)^{(v+1) / 2}}= & \frac{1}{\left(1+\bar{x}^{2} / v\right)^{(v+1) / 2}} \\ & -\frac{v+1}{v} \frac{\bar{x}}{\left(1+\bar{x}^{2} / v\right)^{(v+3) / 2}}(\theta -\bar{x})+R(\theta -\bar{x}) \end{align}

where the remainder, $R(\theta -\bar{x})$ , satisfies $R(\theta -\bar{x}) /(\theta -\bar{x})^{2} \rightarrow 0$ as $\theta \rightarrow \bar{x}$ .

(b)

The remainder in part (a) also satisfies

\int _{-\infty }^{\infty } R(\theta -\bar{x}) e^{-\frac{p}{2 \sigma ^{2}}(\theta -\bar{x})^{2}} d \theta =O\left(1 / p^{3 / 2}\right)

(c)

The numerator and denominator of (5.5.8) can be written

\int _{-\infty }^{\infty } \frac{1}{\left(1+\theta ^{2} / \nu \right)^{(\nu +1) / 2}} e^{-\frac{p}{2 \sigma ^{2}}(\theta -\bar{x})^{2}} d \theta =\frac{\sqrt{2 \pi \sigma ^{2} / p}}{\left(1+\bar{x}^{2} / \nu \right)^{(\nu +1) / 2}}+O\left(\frac{1}{p^{3 / 2}}\right)

and

\begin{align} \int _{-\infty }^{\infty } & \frac{\theta }{\left(1+\theta ^{2} / v\right)^{(v+1) / 2}} e^{-\frac{p}{2 \sigma ^{2}}(\theta -\bar{x})^{2}} d \theta \\ & =\frac{\sqrt{2 \pi \sigma ^{2} / p}}{\left(1+\bar{x}^{2} / v\right)^{(v+1) / 2}}\left[1-\frac{(v+1) / v}{\left(1+\bar{x}^{2} / v\right)}\right] \bar{x}+O\left(\frac{1}{p^{3 / 2}}\right) \end{align}

which yields (5.6.32).

Problem 4.6.16

(a)(b)

For the situation of Example 6.7:

Origin: Stated according to the original source

(a)

Calculate the values of the approximation (5.6.32) for the values of Table 6.2. Are there situations where the estimator (5.6.32) is clearly preferred over the empirical Bayes estimator (5.6.5) as an approximation to the hierarchical Bayes estimator (5.5.8)?

(b)

Extend the argument of Problem 6.15 to calculate the next term in the expansion and, hence, obtain a more accurate approximation to the hierarchical Bayes estimator (5.5.8). For the values of Table 6.2, is this new approximation to (5.5.8) preferable to (5.6.5) and (5.6.32)?

Problem 4.6.17

(a)(b)(c)(d)

Origin: Stated according to the original source

(a)

Show that if $b(\cdot )$ has a bounded second derivative, then

\int b(\lambda ) e^{-n h(\lambda )} d \lambda =b(\hat{\lambda }) \sqrt{\frac{2 \pi }{n h^{\prime \prime }(\hat{\lambda })}} e^{-n h(\hat{\lambda })}+O\left(\frac{1}{n^{3 / 2}}\right)

where $h(\hat{\lambda })$ is the unique minimum of $h(\lambda ), h^{\prime \prime }(\lambda ) \neq 0$ , and $n h(\hat{\lambda }) \rightarrow$ constant as $n \rightarrow \infty$ . [Hint: Expand both $b(\cdot )$ and $h(\cdot )$ in Taylor series around $\hat{\lambda }$ , up to second-order terms. Then, do the term-by-term integration.] This is the Laplace approximation for an integral. For refinements and other developments of this approximation in Bayesian inference, see Tierney and Kadane 1986, Tierney, Kass, and Kadane 1989, and Robert 1994a (Section 9.2.3).

(b)

For the hierarchical model (5.5.1), the posterior mean can be approximated by

\mathbb {E}\left[\Theta \mid x\right]=e^{-n h(\hat{\lambda })}\left[\frac{2 \pi }{n h^{\prime \prime }(\hat{\lambda })}\right]^{1 / 2} \mathbb {E}\left[\Theta \mid x, \hat{\lambda }\right]+O\left(\frac{1}{n^{3 / 2}}\right)

where $h=\frac{1}{n} \log \pi (\lambda \mid x)$ and $\hat{\lambda }$ is the mode of $\pi (\lambda \mid x)$ , the posterior distribution of $\lambda$ .

(c)

If $\pi (\lambda \mid x)$ is the normal distribution with mean $\hat{\lambda }$ and variance $\sigma^{2}=\left[-\left(\partial^{2} / \partial \lambda^{2}\right) \times \left.\log \pi (\lambda \mid x)\right|_{\lambda =\hat{\lambda }}\right]^{-1}$ , then $\mathbb {E}\left[\Theta \mid x\right]=\mathbb {E}\left[\Theta \mid x, \hat{\lambda }\right]+O\left(1 / n^{3 / 2}\right)$ .

(d)

Show that the situation in part (c) arises from the hierarchy

\begin{align} X_{i} \mid \theta _{i} & \sim N\left(\theta _{i}, \sigma ^{2}\right), \\ \theta _{i} \mid \lambda & \sim N\left(\lambda , \tau ^{2}\right), \\ \lambda & \sim \text{ Uniform }(-\infty , \infty ). \end{align}

Problem 4.6.18

(a)(b)

Origin: Stated according to the original source

(a)

Apply the Laplace approximation (5.6.33) to obtain an approximation to the hierarchical Bayes estimator of Example 6.6.

(b)

Compare the approximation from part (a) with the empirical Bayes estimator (5.6.24). Which is a better approximation to the hierarchical Bayes estimator?

Problem 4.6.19

Apply the Laplace approximation (5.6.33) to the hierarchy of Example 6.7 and show that the resulting approximation to the hierarchical Bayes estimator is given by (5.6.32).

Origin: Stated according to the original source

Problem 4.6.20

(a)(b)(c)

Origin: Stated according to the original source

(a)

Verify (6.6.37), that under squared error loss

r(\pi , \delta )=r\left(\pi , \delta ^{\pi }\right)+\mathbb {E}\left[(\delta -\delta ^{\pi })^{2}\right].

(b)

For $X \sim \operatorname {binomial}(p, n), L(p, \delta )=(p-\delta )^{2}$ , and $\pi =\left\{ \pi : \pi =\operatorname {beta}(a, b), a>0\text{, }b>0\right\}$ , determine whether $\hat{p}=x / n$ or $\delta^{0}=\left(a_{0}+x\right) /\left(a_{0}+b_{0}+n\right)$ is more robust, according to (6.6.37).

(c)

Is there an estimator of the form $(c+x) /(c+d+n)$ that you would consider more robust, in the sense of (6.6.37), than either estimator in part (b)?

[In part (b), for fixed $n$ and $\left(a_{0}, b_{0}\right)$ , calculate the Bayes risk of $\hat{p}$ and $\delta^{0}$ for a number of $(a, b)$ pairs.]

Problem 4.6.21

(a)(b)

Origin: Stated according to the original source

(a)

Establish (6.6.39) and (6.6.40) for the class of priors given by (6.6.38).

(b)

Show that the Bayes estimator based on $\pi (\theta ) \in \pi$ in (6.6.38), under squared error loss, is given by (6.6.41).

Section 4.7Risk Comparisons[20/0%]

Problem 4.7.1

(a)(b)

For the situation of Example 7.1:

Origin: Stated according to the original source

(a)

The empirical Bayes estimator of $\boldsymbol {\theta }$ , using an unbiased estimate of $\tau^{2} /\left(\sigma^{2}+\tau^{2}\right)$ , is

\delta ^{\mathrm{EB}}=\left(1-\frac{(p-2) \sigma ^{2}}{|\mathbf{x}|^{2}}\right) \mathbf{x},

the James-Stein estimator.

(b)

The empirical Bayes estimator of $\boldsymbol {\theta }$ , using the marginal MLE of $\tau^{2} /\left(\sigma^{2}+\tau^{2}\right)$ , is

\delta ^{\mathrm{EB}}=\left(1-\frac{p \sigma ^{2}}{|\mathbf{x}|^{2}}\right)^{+} \mathbf{x},

which resembles the positive-part Stein estimator.

Problem 4.7.2

Corollary 7.2 states: let $X \sim N_{p}(\theta , \sigma^{2} I)$ , and let the estimator $\delta$ be of the form $\delta (\mathbf{x})=\mathbf{x}-g(\mathbf{x})$ , where $g(\mathbf{x})=\left\{ g_{i}(\mathbf{x})\right\}$ is differentiable. If $\mathbb {E}_{\theta }\left[\left|(\partial / \partial x_{i}) g_{i}(X)\right|\right]<\infty$ for $i=1, \ldots , p$ , then

R(\theta , \delta )=\mathbb {E}_{\theta }\left[\left|\theta -\delta (X)\right|^{2}\right]=p \sigma ^{2}+\mathbb {E}_{\theta }\left[\left|g(X)\right|^{2}\right]-2 \sigma ^{2} \sum _{i=1}^{p} \mathbb {E}_{\theta }\left[\frac{\partial }{\partial x_{i}} g_{i}(X)\right].

Establish Corollary 7.2. Be sure to verify that the conditions on $g(\mathbf{x})$ are sufficient to allow the integration-by-parts argument. [Stein $(1973,1981)$ develops these representations in the normal case.]

Origin: Stated according to the original source

Problem 4.7.3

(a)(b)(c)

The derivation of an unbiased estimator of the risk (Corollary 7.2) can be extended to a more general model in the exponential family, the model of Corollary 3.3, where $\mathbf{X}=X_{1}, \ldots , X_{p}$ has the density

p_{\boldsymbol {\eta }}(\mathbf{x})=e^{\sum _{i=1}^{p} \eta _{i} x_{i}-A(\boldsymbol {\eta })} h(\mathbf{x}).

Origin: Stated according to the original source

(a)

The Bayes estimator of $\boldsymbol {\eta }$ , under squared error loss, is

\mathbb {E}\left[\eta _{i} \mid \mathbf{x}\right]=\frac{\partial }{\partial x_{i}} \log m(\mathbf{x})-\frac{\partial }{\partial x_{i}} \log h(\mathbf{x}).

Show that the risk of $\mathbb {E}\left[\eta \mid \mathbf{X}\right]]$ has unbiased estimator

\sum _{i=1}^{p}\left[\frac{\partial ^{2}}{\partial x_{i}^{2}}(\log h(\mathbf{x})-2 \log m(\mathbf{x}))+\left(\frac{\partial }{\partial x_{i}} \log m(\mathbf{x})\right)^{2}\right].

[Hint: Theorem 3.5 and Problem 3.4.]

(b)

Show that the risk of the empirical Bayes estimator

\mathbb {E}\left[\eta _{i} \mid \mathbf{x}, \hat{\lambda }\right]=\frac{\partial }{\partial x_{i}} \log m(\mathbf{x} \mid \hat{\lambda }(\mathbf{x}))-\frac{\partial }{\partial x_{i}} \log h(\mathbf{x}).

of Theorem 6.3 has unbiased estimator

\sum _{i=1}^{p}\left[\frac{\partial ^{2}}{\partial x_{i}^{2}}(\log h(\mathbf{x})-2 \log m(\mathbf{x} \mid \hat{\lambda }(\mathbf{x})))+\left(\frac{\partial }{\partial x_{i}} \log m(\mathbf{x} \mid \hat{\lambda }(\mathbf{x}))\right)^{2}\right]

(c)

Use the results of part (b) to derive an unbiased estimator of the risk of the positivepart Stein estimator of (7.7.10).

Problem 4.7.4

Verify (7.7.9), the expression for the Bayes risk of $\delta^{\tau_{0}}$ . (Problem 3.12 may be helpful.)

Origin: Stated according to the original source

Problem 4.7.5

(a)(b)

A general version of the empirical Bayes estimator (7.7.3) is given by

\delta ^{c}(\mathbf{x})=\left(1-\frac{c \sigma ^{2}}{|\mathbf{x}|^{2}}\right) \mathbf{x},

where $c$ is a positive constant.

Origin: Stated according to the original source

(a)

Use Corollary 7.2 to verify that

\mathbb {E}_{\theta }\left[\left|\boldsymbol {\theta }-\delta ^{c}(\mathbf{X})\right|^{2}\right]=p \sigma ^{2}+c \sigma ^{4}[c-2(p-2)] \mathbb {E}_{\theta }\left[\frac{1}{|\mathbf{X}|^{2}}\right].

(b)

Show that the Bayes risk, under $\Theta \sim N_{p}\left(0, \tau^{2} I\right)$ , is given by

r\left(\pi , \delta ^{c}\right)=\sigma ^{2}\left[p+\frac{c \sigma ^{2}}{\sigma ^{2}+\tau ^{2}}\left(\frac{c}{p-2}-2\right)\right]

and is minimized by choosing $c=p-2$ .

Problem 4.7.6

(a)(b)(c)

For the model

\begin{align} \mathbf{X} \mid \boldsymbol {\theta } & \sim N_{p}\left(\boldsymbol {\theta }, \sigma ^{2} I\right), \\ \boldsymbol {\theta } \mid \tau ^{2} & \sim N_{p}\left(\mu , \tau ^{2} I\right): \end{align}

Show that:

Origin: Stated according to the original source

(a)

The empirical Bayes estimator, using an unbiased estimator of $\tau^{2} /\left(\sigma^{2}+\tau^{2}\right)$ , is the Stein estimator

\delta _{i}^{\mathrm{JS}}(\mathbf{x})=\mu _{i}+\left(1-\frac{(p-2) \sigma ^{2}}{\Sigma \left(x_{i}-\mu _{i}\right)^{2}}\right)\left(x_{i}-\mu _{i}\right).

(b)

If $p \geq 3$ , the Bayes risk, under squared error loss, of $\delta^{\mathrm{JS}}$ is $r\left(\tau , \delta^{\mathrm{JS}}\right)=r\left(\tau , \delta^{\tau }\right)+ 2 \sigma^{4} /\left(\sigma^{2}+\tau^{2}\right)$ , where $r\left(\tau , \delta^{\tau }\right)$ is the Bayes risk of the Bayes estimator.

(c)

If $p<3$ , the Bayes risk of $\delta^{\mathrm{JS}}$ is infinite. [Hint: Show that if $Y \sim \chi_{m}^{2}, \mathbb {E}\left[1 / Y\right]< \infty \Longleftrightarrow m<3]$ .

Problem 4.7.7

For the model

\begin{align} \mathbf{X} \mid \boldsymbol {\theta } & \sim N_{p}\left(\boldsymbol {\theta }, \sigma ^{2} I\right), \\ \boldsymbol {\theta } \mid \tau ^{2} & \sim N\left(\mu , \tau ^{2} I\right) \end{align}

the Bayes risk of the ordinary Stein estimator

\delta _{i}(\mathbf{x})=\mu _{i}+\left(1-\frac{(p-2) \sigma ^{2}}{\Sigma \left(x_{i}-\mu _{i}\right)^{2}}\right)\left(x_{i}-\mu _{i}\right)

is uniformly larger than its positive-part version

\delta _{i}^{+}(\mathbf{x})=\mu _{i}+\left(1-\frac{(p-2) \sigma ^{2}}{\Sigma \left(x_{i}-\mu _{i}\right)^{2}}\right)^{+}\left(x_{i}-\mu _{i}\right).

Origin: Stated according to the original source

Problem 4.7.8

(a)(b)(c)

Theorem 7.5 holds in greater generality than just the normal distribution. Suppose $\mathbf{X}$ is distributed according to the multivariate version of the exponential family $p_{\eta }(x)$ of (33.7),

p_{\eta }(\mathbf{x})=e^{\eta ^{\prime } \mathbf{x}-A(\eta )} h(\mathbf{x}), \quad -\infty <x_{i}<\infty ,

and a multivariate conjugate prior distribution [generalizing (3.19)] is used.

Origin: Stated according to the original source

(a)

Show that $\mathbb {E}\left[\mathbf{X} \mid \eta \right]=\nabla A(\eta )$ .

(b)

If $\mu =0$ in the prior distribution (see 3.19), show that $r(\tau , \delta ) \geq r\left(\tau , \delta^{+}\right)$ , where $\delta (\mathbf{x})=[1-B(\mathbf{x})] \mathbf{x}$ and $\delta^{+}(\mathbf{x})=[1-B(\mathbf{x})]^{+} \mathbf{x}$ .

(c)

If $\mu \neq 0$ , the estimator $\delta (\mathbf{x})$ would be modified to $\mu +\delta (\mathbf{x}-\mu )$ . Establish a result similar to part (b) for this estimator.

Problem 4.7.9

(a)(b)

Origin: Stated according to the original source

(a)

For the model (7.7.15), show that the marginal distribution of $X_{i}$ is negative binomial $(a, 1 / b+1)$ ; that is,

\mathbb {P}\left(X_{i}=x\right)=\binom {a+x-1}{x}\left(\frac{b}{b+1}\right)^{x}\left(\frac{1}{b+1}\right)^{a}

with $\mathbb {E}\left[X_{i}\right]=a b$ and var $X_{i}=a b(b+1)$ .

(b)

If $X_{1}, \ldots , X_{m}$ are iid according to the negative binomial distribution in part (a), show that the conditional distribution of $X_{j} \mid \sum_{1}^{m} X_{i}$ is the negative hypergeometric distribution, given by

\mathbb {P}\left(X_{j}=x \mid \sum _{1}^{m} X_{i}=t\right)=\frac{\binom {a+x-1}{x}\binom {(m-1) a+t-x-1}{t-x}}{\binom {m a+t-1}{t}}

with $\mathbb {E}\left[X_{j}\right]=t / m$ and var $X_{j}=(m-1) t(m a+t) / m^{2}(m a+1)$ .

Problem 4.7.10

(a)(b)(c)

For the situation of Example 7.6:

Origin: Stated according to the original source

(a)

Show that the Bayes estimator under the loss $L_{k}(\lambda , \delta )$ of (7.7.16) is given by (7.7.17).

(b)

Verify (7.7.19) and (7.7.20).

(c)

Evaluate the Bayes risks $r\left(0, \delta^{1}\right)$ and $r\left(1, \delta^{0}\right)$ . Which estimator, $\delta^{0}$ or $\delta^{1}$ , is more robust?

Problem 4.7.11

For the situation of Example 7.6, evaluate the Bayes risk of the empirical Bayes estimator (7.7.20) for $k=0$ and 1. What values of the unknown hyperparameter $b$ are least and which are most favorable to the empirical Bayes estimator?

Origin: Stated according to the original source

Problem 4.7.12

Consider a hierarchical Bayes estimator for the Poisson model (7.7.15) with loss (7.7.16). Using the distribution (5.6.27) for the hyperparameter $b$ , show that the Bayes estimator is

\left(\frac{p \bar{x}+\alpha -k}{p \bar{x}+p a+\alpha +\beta -k}\right)\left(a+x_{i}-k\right).

[Hint: Show that the Bayes estimator is $\mathbb {E}\left[\lambda^{1-k} \mid \mathbf{x}\right] / \mathbb {E}\left[\lambda^{-k} \mid \mathbf{x}\right]$ and that

\mathbb {E}\left[\lambda ^{r} \mid \mathbf{x}\right]=\frac{\Gamma (p \bar{x}+p a+\alpha +\beta ) \Gamma (p \bar{x}+\alpha +r)}{\Gamma (p \bar{x}+\alpha ) \Gamma (p \bar{x}+p a+\alpha +\beta +r)} \frac{\Gamma \left(a+x_{i}+r\right)}{\Gamma \left(a+x_{i}\right)}.

Origin: Stated according to the original source

Problem 4.7.13

(a)(b)

Prove the following: Two matrix results that are useful in calculating estimators from multivariate hierarchical models are

Origin: Stated according to the original source

(a)

For any vector $a$ of the form $a=\left(I-\frac{1}{s} J\right) b, \mathbf{1}^{\prime } a=\Sigma a_{i}=0$ .

(b)

If $B$ is an idempotent matrix (that is, $B^{2}=I$ ) and $a$ is a scalar, then

(I+a B)^{-1}=I-\frac{a}{1+a} B.

Problem 4.7.14

(a)(b)(c)(d)

For the situation of Example 7.7:

Origin: Stated according to the original source

(a)

Show how to derive the empirical Bayes estimator $\delta^{L}$ of (7.7.28).

(b)

Verify the Bayes risk of $\delta^{L}$ of (7.7.29).

For the situation of Example 7.8:

(c)

Show how to derive the empirical Bayes estimator $\delta^{\mathrm{EB}_{2}}$ of (7.7.33).

(d)

Verify the Bayes risk of $\delta^{\mathrm{EB}_{2}}$ , (7.7.34).

Problem 4.7.15

(a)(b)(c)

The empirical Bayes estimator (7.7.27) can also be derived as a hierarchical Bayes estimator. Consider the hierarchical model

\begin{align} X_{i j} \mid \xi _{i} & \sim N\left(\xi _{i}, \sigma ^{2}\right), \quad j=1, \ldots , n, \quad i=1, \ldots , s, \\ \xi _{i} \mid \mu & \sim N\left(\mu , \tau ^{2}\right), \quad i=1, \ldots , s, \\ \mu & \sim \text{ Uniform }(-\infty , \infty ) \end{align}

where $\sigma^{2}$ and $\tau^{2}$ are known.

Origin: Stated according to the original source

(a)

Show that the Bayes estimator, with respect to squared error loss, is

\mathbb {E}\left[\xi _{i} \mid \mathbf{x}\right]=\frac{\sigma ^{2}}{\sigma ^{2}+n \tau ^{2}} \mathbb {E}\left[\mu \mid \mathbf{x}\right]+\frac{n \tau ^{2}}{\sigma ^{2}+n \tau ^{2}} \bar{x}_{i}

where $\mathbb {E}\left[\mu \mid \mathbf{x}\right]$ is the posterior mean of $\mu$ .

(b)

Establish that $\mathbb {E}\left[\mu \mid \mathbf{x}\right]=\bar{x}=\Sigma x_{i j} / n s$ . [This can be done by evaluating the expectation directly, or by showing that the posterior distribution of $\xi_{i} \mid \mathbf{x}$ is

\xi _{i} \left\lvert \, \mathbf{x} \sim N\left[\frac{\sigma ^{2}}{\sigma ^{2}+n \tau ^{2}} \bar{x}+\frac{n \tau ^{2}}{\sigma ^{2}+n \tau ^{2}} \bar{x}_{i}, \frac{\sigma ^{2}}{\sigma ^{2}+n \tau ^{2}}\left(n \tau ^{2}+\frac{\sigma ^{2}}{s}\right)\right].\right.

Note that the $\xi_{i}$ 's are not independent a posteriori. In fact,

\boldsymbol {\xi } \left\lvert \, \mathbf{x} \sim N_{s}\left(\frac{n \tau ^{2}}{\sigma ^{2}+n \tau ^{2}} M, \frac{n \sigma ^{2} \tau ^{2}}{\sigma ^{2}+n \tau ^{2}} M\right)\right.,

where $\left.M=I+\left(\sigma^{2} / n \tau^{2}\right) J\right]$ .

(c)

Show that the empirical Bayes estimator (7.7.32) can also be derived as a hierarchical Bayes estimator, by appending the specification $(\alpha , \beta ) \sim$ Uniform $\left(\Re^{2}\right)$ [that is, $\pi (\alpha , \beta )=d \alpha d \beta ,-\infty <\alpha , \beta <\infty$ ] to the hierarchy (7.7.30).

Problem 4.7.16

(a)(b)(c)

Generalization of model (7.7.23) to the case of unequal $n_{i}$ is, perhaps, not as straightforward as one might expect. Consider the generalization

\begin{align} X_{i j} \mid \xi _{i} & \sim N\left(\xi _{i}, \sigma ^{2}\right), \quad j=1, \ldots , n_{i}, \quad i=1, \ldots , s, \\ \xi _{i} \mid \mu & \sim N\left(\mu , \tau _{i}^{2}\right), \quad i=1, \ldots , s. \end{align}

We also make the assumption that $\tau_{i}^{2}=\tau^{2} / n_{i}$ . Show that:

Origin: Stated according to the original source

(a)

The above model is equivalent to

\begin{align} & \mathbf{Y} \sim N_{s}\left(\lambda , \sigma ^{2} I\right), \\ & \lambda \sim N_{s}\left(\mathbf{Z} \mu , \tau ^{2} I\right) \end{align}

where $Y_{i}=\sqrt{n_{i}} \bar{X}_{i}, \lambda_{i}=\sqrt{n_{i}} \xi_{i}$ and $\mathbf{z}=\left(\sqrt{n_{1}}, \ldots , \sqrt{n_{s}}\right)^{\prime }$ .

(b)

The Bayes estimator of $\xi_{i}$ , using squared error loss, is

\frac{\sigma ^{2}}{\sigma ^{2}+\tau ^{2}} \mu +\frac{\tau ^{2}}{\sigma ^{2}+\tau ^{2}} \bar{x}_{i}.

(c)

The marginal distribution of $Y_{i}$ is $Y_{i} \sim N_{s}\left(\mathbf{Z} \mu ,\left(\sigma^{2}+\tau^{2}\right) I\right)$ , and an empirical Bayes estimator of $\xi$ is

\delta _{i}^{E B}=\bar{x}+\left(1-\frac{(s-3) \sigma ^{2}}{\Sigma n_{i}\left(\bar{x}_{i}-\bar{x}\right)^{2}}\right)\left(\bar{x}_{i}-\bar{x}\right)

where $\bar{x}_{i}=\Sigma_{j} x_{i j} / n_{i}$ and $\bar{x}=\Sigma_{i} n_{i} \bar{x}_{i} / \Sigma_{i} n_{i}$ . [Without the assumption that $\tau_{i}^{2}=\tau^{2} / n_{i}$ , one cannot get a simple empirical Bayes estimator. If $\tau_{i}^{2}=\tau^{2}$ , the likelihood estimation can be used to get an estimate of $\tau^{2}$ to be used in the empirical Bayes estimator. This is discussed by Morris (1983a).]

Problem 4.7.17

(a)(b)(c)(d)

(Empirical Bayes estimation in a general case). A general version of the hierarchical models of Examples 7.7 and 7.8 is

\begin{align} & \mathbf{X} \mid \xi \sim N_{s}\left(\xi , \sigma ^{2} I\right), \\ & \xi \mid \boldsymbol {\beta } \sim N_{s}\left(\mathbf{Z} \boldsymbol {\beta }, \tau ^{2} I\right) \end{align}

where $\sigma^{2}$ and $\mathbf{Z}_{s \times r}$ , of rank $r$ , are known and $\tau^{2}$ and $\boldsymbol {\beta }_{r \times 1}$ are unknown. Under this model show that:

Origin: Stated according to the original source

(a)

The Bayes estimator of $\xi$ , under squared error loss, is

\mathbb {E}\left[\boldsymbol {\xi } \mid \mathbf{x}, \boldsymbol {\beta }\right]=\frac{\sigma ^{2}}{\sigma ^{2}+\tau ^{2}} z \boldsymbol {\beta }+\frac{\tau ^{2}}{\sigma ^{2}+\tau ^{2}} \mathbf{x}.

(b)

Marginally, the distribution of $\mathbf{X} \mid \boldsymbol {\beta }$ is $\mathbf{X} \mid \boldsymbol {\beta } \sim N_{s}\left(\mathbf{Z} \boldsymbol {\beta },\left(\sigma^{2}+\tau^{2}\right) I\right)$ .

(c)

Under the marginal distribution in part (b),

\begin{align} \mathbb {E}\left[\left(\mathbf{Z}^{\prime } \mathbf{Z}\right)^{-1} \mathbf{Z}^{\prime } \mathbf{x}\right] & =\mathbb {E}\left[\hat{\boldsymbol {\beta }}\right]=\boldsymbol {\beta }, \\ \mathbb {E}\left[\frac{s-r-2}{|\mathbf{X}-\mathbf{Z} \hat{\boldsymbol {\beta }}|^{2}}\right] & =\frac{1}{\sigma ^{2}+\tau ^{2}}, \end{align}

and, hence, an empirical Bayes estimator of $\xi$ is

\delta ^{E B}=\mathbf{Z} \hat{\boldsymbol {\beta }}+\left(1-\frac{(s-r-2) \sigma ^{2}}{|\mathbf{x}-\mathbf{Z} \hat{\boldsymbol {\beta }}|^{2}}\right)(\mathbf{x}-\mathbf{Z} \hat{\boldsymbol {\beta }}).

(d)

The Bayes risk of $\delta^{E B}$ is $r\left(\tau , \delta^{\tau }\right)+(r+2) \sigma^{4} /\left(\sigma^{2}+\tau^{2}\right)$ , where $r\left(\tau , \delta^{\tau }\right)$ is the risk of the Bayes estimator.

Problem 4.7.18

(a)(b)(c)

In a manner similar to the previous problem, we can derive hierarchical Bayes estimators for the model

\begin{align} \mathbf{X} \mid \boldsymbol {\xi } & \sim N_{s}\left(\boldsymbol {\xi }, \sigma ^{2} I\right), \\ \boldsymbol {\xi } \mid \boldsymbol {\beta } & \sim N_{s}\left(\mathbf{Z} \boldsymbol {\beta }, \tau ^{2} I\right), \\ \boldsymbol {\beta } & \sim \operatorname {Uniform}\left(\Re ^{r}\right) \end{align}

where $\sigma^{2}$ and $\mathbf{Z}_{s \times r}$ , of rank $r$ , are known and $\tau^{2}$ is unknown.

Origin: Stated according to the original source

(a)

The prior distribution of $\boldsymbol {\xi }$ , unconditional on $\beta$ , is proportional to

\pi (\xi )=\int _{\Re ^{r}} \pi (\xi \mid \beta ) d \beta \propto e^{-\frac{1}{2} \frac{\xi ^{\prime }(I-H) \xi }{\tau ^{2}}},

where $H=\mathbf{Z}\left(\mathbf{Z}^{\prime } \mathbf{Z}\right)^{-1} \mathbf{Z}^{\prime }$ projects from $\Re^{s}$ to $\Re^{r}$ . [Hint: Establish that

\begin{align} (\xi -\mathbf{Z} \boldsymbol {\beta })^{\prime }(\boldsymbol {\xi }-\mathbf{Z} \boldsymbol {\beta })=\xi ^{\prime }(I-H) \boldsymbol {\xi } & \\ & +\left[\boldsymbol {\beta }-\left(\mathbf{Z}^{\prime } \mathbf{Z}\right)^{-1} \mathbf{Z}^{\prime } \boldsymbol {\xi }\right]^{\prime } \mathbf{Z}^{\prime } \mathbf{Z}\left[\boldsymbol {\beta }-\left(\mathbf{Z}^{\prime } \mathbf{Z}\right)^{-1} \mathbf{Z}^{\prime } \boldsymbol {\xi }\right] \end{align}

to perform the integration on $\boldsymbol {\beta }$ .]

(b)

Show that

\boldsymbol {\xi } \left\lvert \, \mathbf{x} \sim N_{s}\left(\frac{\tau ^{2}}{\sigma ^{2}+\tau ^{2}} M, \frac{\sigma ^{2} \tau ^{2}}{\sigma ^{2}+\tau ^{2}} M\right)\right.

where $M=I+\left(\sigma^{2} / \tau^{2}\right) H$ , and hence that the Bayes estimator is given by

\frac{\sigma ^{2}}{\sigma ^{2}+\tau ^{2}} H \mathbf{x}+\frac{\tau ^{2}}{\sigma ^{2}+\tau ^{2}} \mathbf{x}

where $\mathbf{Z} \hat{\boldsymbol {\beta }}=H \mathbf{x}$ . [Hint: Establish that

\begin{align} & \frac{1}{\tau ^{2}} \xi ^{\prime }(I-H) \xi +\frac{1}{\sigma ^{2}}(\mathbf{x}-\xi )^{\prime }(\mathbf{x}-\xi ) \\ & = \frac{\sigma ^{2}+\tau ^{2}}{\sigma ^{2} \tau ^{2}}\left[\left(\xi -\frac{\tau ^{2}}{\sigma ^{2}+\tau ^{2}} M \mathbf{x}\right)^{\prime } M^{-1}\left(\xi -\frac{\tau ^{2}}{\sigma ^{2}+\tau ^{2}} M \mathbf{x}\right)\right] \\ & +\frac{1}{\sigma ^{2}+\tau ^{2}} \mathbf{x}^{\prime }(I-H) \mathbf{x} \end{align}

where $M^{-1}=I-\frac{\sigma^{2}}{\sigma^{2}+\tau^{2}} H$ .]

(c)

Marginally, $\mathbf{X}^{\prime }(I-H) \mathbf{X} \sim \left(\sigma^{2}+\tau^{2}\right) \chi_{s-r}^{2}$ . This leads us to the empirical Bayes estimator

H \mathbf{x}+\left(1-\frac{(s-r-2) \sigma ^{2}}{\mathbf{x}^{\prime }(I-H) \mathbf{x}}\right)(\mathbf{x}-H \mathbf{x})

which is equal to the empirical Bayes estimator of Problem 7.17(c).

[The model in this and the previous problem can be substantially generalized. For example, both $\sigma^{2} I$ and $\tau^{2} I$ can be replaced by full, positive definite matrices. At the cost of an increase in the complexity of the matrix calculations and the loss of simple answers, hierarchical and empirical Bayes estimators can be computed. The covariances, either scalar or matrix, can also be unknown, and inverted gamma (or inverted Wishart) prior distributions can be accommodated. Calculations can be implemented via the Gibbs sampler. Note that these generalizations encompass the "unequal $n_{i}$ " case (see Problem 7.16), but there are no simple solutions for this case. Many of these estimators also possess a minimax property, which will be discussed in Chapter 5.]

Problem 4.7.19

As noted by Morris (1983a), an analysis of variance-type hierarchical model, with unequal $n_{i}$ , will yield closed-form empirical Bayes estimators if the prior variances are proportional to the sampling variances. Show that, for the model

\begin{align} X_{i j} \mid \xi _{i} & \sim N\left(\xi _{i}, \sigma ^{2}\right), \quad j=1, \ldots , n_{i}, \quad i=1, \ldots , s \\ \boldsymbol {\xi } \mid \boldsymbol {\beta } & \sim N_{s}\left(\mathbf{Z} \boldsymbol {\beta }, \tau ^{2} D^{-1}\right) \end{align}

where $\sigma^{2}$ and $\mathbf{Z}_{s \times r}$ , of full rank $r$ , are known, $\tau^{2}$ is unknown, and $D=\operatorname {diag}\left(n_{1}, \ldots , n_{s}\right)$ , an empirical Bayes estimator is given by

\delta ^{\mathrm{EB}}=\mathbf{Z} \hat{\boldsymbol {\beta }}+\left(1-\frac{(s-r-2) \sigma ^{2}}{(\overline{\mathbf{x}}-\mathbf{Z} \hat{\boldsymbol {\beta }})^{\prime } D(\overline{\mathbf{x}}-\mathbf{Z} \hat{\boldsymbol {\beta }})}\right)(\overline{\mathbf{x}}-\mathbf{Z} \hat{\boldsymbol {\beta }})

with $\overline{\mathbf{x}}_{i}=\Sigma_{j} x_{i j} / n_{i}, \overline{\mathbf{x}}=\left\{ \bar{x}_{i}\right\}$ , and $\hat{\boldsymbol {\beta }}=\left(\mathbf{Z}^{\prime } D \mathbf{Z}\right)^{-1} \mathbf{Z}^{\prime } D \overline{\mathbf{x}}$ .

Origin: Stated according to the original source

Problem 4.7.20

(a)(b)(c)(d)

An entertaining (and unjustifiable) result which abuses a hierarchical Bayes calculation yields the following derivation of the James-Stein estimator. Let $X \sim N_{p}(\boldsymbol {\theta }, I)$ and $\boldsymbol {\theta } \mid \tau^{2} \sim N_{p}\left(0, \tau^{2} I\right)$ .

Origin: Stated according to the original source

(a)

Verify that conditional on $\tau^{2}$ , the posterior and marginal distributions are given by

\begin{align} \pi \left(\boldsymbol {\theta } \mid \mathbf{x}, \tau ^{2}\right) & =N_{p}\left(\frac{\tau ^{2}}{\tau ^{2}+1} \mathbf{x}, \frac{\tau ^{2}}{\tau ^{2}+1} I\right) \\ m\left(\mathbf{x} \mid \tau ^{2}\right) & =N_{p}\left[0,\left(\tau ^{2}+1\right) I\right] \end{align}

(b)

Show that, taking $\pi \left(\tau^{2}\right)=1,-1<\tau^{2}<\infty$ , we have

\begin{align} & \iint _{\Re p} \boldsymbol {\theta } \pi \left(\boldsymbol {\theta } \mid \mathbf{x}, \tau ^{2}\right) m\left(\mathbf{x} \mid \tau ^{2}\right) d \boldsymbol {\theta } d \tau ^{2} \\ & =\frac{\mathbf{x}}{(2 \pi )^{p / 2}\left(|\mathbf{x}|^{2}\right)^{p / 2-1}}\left[\Gamma \left(\frac{p-2}{2}\right) 2^{(p-2) / 2}-\frac{\Gamma (p / 2) 2^{p / 2}}{|\mathbf{x}|^{2}}\right] \end{align}

and

\begin{align} & \iint _{\Re p} \pi \left(\boldsymbol {\theta } \mid \mathbf{x}, \tau ^{2}\right) m\left(\mathbf{x} \mid \tau ^{2}\right) d \boldsymbol {\theta } d \tau ^{2} \\ & =\frac{1}{(2 \pi )^{p / 2}\left(|\mathbf{x}|^{2}\right)^{p / 2-1}} \Gamma \left(\frac{p-2}{2}\right) 2^{(p-2) / 2} \end{align}

and hence

\mathbb {E}\left[\boldsymbol {\theta } \mid \mathbf{x}\right]=\left(1-\frac{p-2}{|\mathbf{x}|^{2}}\right) \mathbf{x}.

(c)

Explain some implications of the result in part (b) and, why it cannot be true. [Try to reconcile it with (3.3.12).]

(d)

Why are the calculations in part (b) unjustified?