Hierarchical Bayes

$\mathbb {E}\left[\theta \mid x\right]=E[\mathbb {E}\left[\theta \mid x, \lambda \right]]$;

$\operatorname {var}(\theta \mid x)=E[\operatorname {var}(\theta \mid x, \lambda )]+\operatorname {var}[\mathbb {E}\left[\theta \mid x, \lambda \right]]$;

The marginal prior of $\theta$, unconditional on $\tau^{2}$, is given by

which for $a=v / 2$ and $b=2 / v$ is Student's $t$-distribution with $v$ degrees of freedom.

The marginal posterior of $\tau^{2}$ is given by

Show that

[Note that $\mathbb {E}\left[\eta \mid \mathbf{x}\right]$ and $\operatorname {var}(\eta \mid \mathbf{x})$ are not expressible in a simple form.]

Unconditionally on $\eta$, the $\theta_{i}$ 's have conditional covariance

Ignoring the prior distribution of $\eta$, show how to construct an empirical Bayes estimator of $\theta_{i}$. (Again, this is not expressible in a simple form.)

Analogous to Problem 1.7.9, establish that for any random variable $X, Y$, and $Z$,

For the hierarchy

show that $\operatorname {cov}\left(\Theta_{i}, \Theta_{j} \mid \mathbf{x}\right)=\operatorname {cov}\left[\mathbb {E}\left[\Theta_{i} \mid \mathbf{x}, \lambda \right], \mathbb {E}\left[\Theta_{j} \mid \mathbf{x}, \lambda \right]\right]$.

If $\mathbb {E}\left[\Theta_{i} \mid \mathbf{x}, \lambda \right]=g\left(x_{i}\right)+h(\lambda ), i=1, \ldots , p$, where $g(\cdot )$ and $h(\cdot )$ are known, then

[Part (c) points to what can be considered a limitation in the applicability of some hierarchical models, that they imply a positive correlation structure in the posterior distribution.]

Using the prior distribution for $\gamma (b)$ given in (5.6.27), show that the mode of the posterior distribution $\pi (b \mid \mathbf{x})$ is $\hat{b}=(p \bar{x}+\alpha -1) /(p a+\beta -1)$, and hence the empirical Bayes estimator based on this $\hat{b}$ does not equal the hierarchical Bayes estimator (5.6.29).

Show that if we estimate $b /(b+1)$ using its posterior expectation $E[b /(b+1) \mid \mathbf{x}]$, then the resulting empirical Bayes estimator is equal to the hierarchical Bayes estimator.

To calculate $\int h(x) f_{X}(x) d x$, generate a sample $X_{1}, \ldots , X_{m}$, iid, from $f_{X}(x)$. Then, $1 / m \sum_{i=1}^{m} h\left(x_{i}\right) \rightarrow \int h(x) f_{X}(x) d x$ as $m \rightarrow \infty$.

If it is difficult to generate random variable from $f_{X}(x)$, then generate pairs of random variables

Then, $1 / m \sum_{i=1}^{m} h\left(x_{i}\right) \rightarrow \int h(x) f_{X}(x) d x$ as $m \rightarrow \infty$. [Show that if $X$ is generated according to $Y \sim f_{Y}(y)$ and $X \sim f_{X \mid Y}(x \mid Y)$, then $\mathbb {P}\left(X \leq a\right)=\int_{-\infty }^{a} f_{x}(x) d x$.]

If it is difficult to generate as in part (b), then generate

for $i=1, \ldots , K$ and $m=1, \ldots , M$.

Show that:

1. for each $m,\left\{ X_{m_{i}}\right\}$ is a Markov chain. If it is also an ergodic Markov chain $X_{m_{i}} \xrightarrow {\mathcal{L}} X$, as $i \rightarrow \infty$, where $X$ has the stationary distribution of the chain.

2. If the stationary distribution of the chain is $f_{X}(x)$, then

   $$\frac{1}{M} \sum _{m=1}^{M} h\left(x_{m_{k}}\right) \rightarrow \int h(x) f_{X}(x) d x$$

   as $K, M \rightarrow \infty$.

Show that the function $h^{*}(\cdot )$ that solves $h^{*}(x)=\int k\left(x, x^{\prime }\right) h^{*}\left(x^{\prime }\right) d x^{\prime }$ is $h^{*}(x)= f_{X}(x)$, the marginal distribution of $X$.

Write down the analogous integral equation that is solved by $f_{Y}(y)$.

Define a sequence of functions recursively by $h_{i+1}(x)=\int k\left(x, x^{\prime }\right) h_{i}\left(x^{\prime }\right) d x_{I}^{\prime }$ where $h_{0}(x)$ is arbitrary but satisfies $\sup_{x}\left|\frac{h_{0}(x)}{h^{*}(x)}\right|<\infty$. Show that

and, hence, $h_{i}(x)$ converges to $h^{*}(x)$.

To calculate $h_{i+1}$ using Monte Carlo integration:

1. Generate $X_{j}^{\prime } \sim h_{i}\left(x^{\prime }\right), \quad j=1, \ldots , J$.

2. Generate, for each $x_{j}^{\prime }, Y_{j k} \sim f_{Y \mid X}\left(y \mid x_{j}^{\prime }\right), k=1, \ldots , K$.

3. Calculate $\hat{h}_{i+1}(x)=\frac{1}{J} \sum_{j=1}^{J} \frac{1}{K} \sum_{k=1}^{K} f_{X \mid Y}\left(x \mid y_{j k}\right)$. Then, $\hat{h}_{i+1}(x) \rightarrow h_{i+1}(x)$ as $J, K \rightarrow \infty$, and hence the data augmentation algorithm converges.

To implement (a)(i), we must be able to generate a random variable from a mixture distribution. Show that if $f_{Y}(y)=\sum_{i=1}^{n} a_{i} g_{i}(y), \Sigma a_{i}=1$, then the algorithm

1. Select $g_{i}$ with probability $a_{i}$

2. Generate $Y \sim g_{i}$ produces a random variable with distribution $f_{Y}$.

Show that $h_{M}(x) \rightarrow f_{X}(x)$ as $M \rightarrow \infty$.

Given $x_{m_{k}}$, a Monte Carlo approximation to $h_{M}(x)$ is

where $Y_{k_{j}} \sim f_{Y \mid X}\left(y \mid x_{m_{k}}\right)$ and $\hat{h}_{M}(x) \rightarrow h_{M}(x)$ as $J \rightarrow \infty$.

Hence, as $M, J \rightarrow \infty , \hat{h}_{M}(x) \rightarrow f_{X}(x)$.

Discuss when equality might hold in (b). Can you give an example?

Show that $-\log U \sim \exp (1)$.

Show that $-\sum_{i=1}^{n} \log U_{i} \sim \operatorname {Gamma}(n, 1)$, where $U_{1}, \ldots , U_{n}$ are iid as $U(0,1)$.

Let $X \sim \operatorname {Exp}(a, b)$. Write $X$ as a function of $U$.

Let $X \sim \operatorname {Gamma}(n, \beta )$, $n$ an integer. Write $X$ as a function of $U_{1}, \ldots , U_{n}$, iid as $U(0,1)$.

Show that the algorithm will generate $X \sim f(x)$.

Starting with $Y \sim \exp (1)$, show how to generate $X \sim N(0,1)$.

Show how to generate a gamma random variable with a noninteger shape parameter.

Show that the posterior distribution of $\theta_{i}(1 \leq i \leq k-1)$ is

where $\tau_{i}^{2}=\left(\Sigma_{1}^{i} \sigma_{j}^{2}\right)\left(\Sigma_{i+1}^{k} \sigma_{j}^{2}\right) / \Sigma_{i}^{k} \sigma_{j}^{2}$ and $\alpha_{i}=\tau_{i}^{2} / \Sigma_{1}^{i} \sigma_{j}^{2}$.

Find an expression for the Kullback-Leibler information $K\left[\pi \left(\theta_{i} \mid x, \theta_{k}\right), \pi \left(\theta_{i} \mid \theta_{k}\right)\right]$ and show that it is a decreasing function of $i$.

Show that $R_{\alpha }(f, g)$ satisfies $R_{\alpha }(f, g)>0$ and $R_{\alpha }(f, f)=0$.

Show that Theorem 5.7 holds if $R_{\alpha }(f, g)$ is used instead of $K[f, g]$.

Show that $\lim_{\alpha \rightarrow 1} R_{\alpha }(f, g)=K[f, g]$, and provide another proof of Theorem 5.7.

Show that the Jeffreys prior is $\pi_{J}(\tau , \omega )=(1-\tau )^{-1 / 2} \omega^{-1 / 2}(1-\omega )^{-1 / 2}$.

If $\tau$ is considered a nuisance parameter, the reference prior is

Compare the posterior means $\mathbb {E}\left[\omega \mid r, s, m\right]$ under both the Jeffreys and reference priors. Is one more appropriate?

What is the effect of the different priors on the posterior variance?

If integration and differentiation can be interchanged, show that

where $I(\theta )$ is the Fisher information of $f(x \mid \theta )$.

George and McCulloch (1993) argue that choosing $\pi (\theta )=(\operatorname {det}[\nabla \nabla \psi (\theta , \theta )])^{1 / 2}$ is an appealing least informative choice of priors. What justification can you give for this?