Estimation in Parametric Models

Show that the priors in the following cases are conjugate priors:

In Exercise 4.1, find the posterior mean and variance for each case.

Let $X=(X_1,\ldots ,X_n)$ be a random sample from the uniform distribution on the interval $(0,\theta )$, where $\theta >0$ is unknown. Let the prior of $\theta$ be the log-normal distribution with parameter $(\mu_0,\sigma_0^2)$, where $\mu_0\in \mathbb {R}$ and $\sigma_0>0$ are known constants.

Let $\overline{X}$ be the sample mean of a random sample of size $n$ from $\mathscr {N}\left(\theta , \sigma^2\right)$ with a known $\sigma >0$ and an unknown $\theta \in \mathbb {R}$. Let $\pi (\theta )$ be a prior density with respect to a $\sigma$-finite measure $\nu$ on $\mathbb {R}$.

Let $X=(X_1,\ldots ,X_n)$ be a random sample from $P$ with discrete probability density $f_{\theta ,j}$, where $\theta \in (0,1)$, $j=1,2$, $f_{\theta ,1}$ is the Poisson distribution with mean $\theta$, and $f_{\theta ,2}$ is the binomial distribution with size $1$ and probability $\theta$. Consider the estimation of $\theta$ under the squared error loss. Suppose that the prior of $\theta$ is the uniform distribution on $(0,1)$, the prior of $j$ is $\mathbb {P}\left(j=1\right)=\mathbb {P}\left(j=2\right)=\frac12$, and the joint prior of $(\theta ,j)$ is the product probability of the two marginal priors. Show that the Bayes action is

where $x=(x_1,\ldots ,x_n)$ is the vector of observations, $t=x_1+\cdots +x_n$, $B(t)=\int_0^1\theta^t(1-\theta )^{n-t}\, d\theta$, $G(t)=\int_0^1\theta^te^{-n\theta }\, d\theta$, and $H(x)$ is a function of $x$ with range $\left\{ 0,1\right\}$.

Let $X$ be a sample from $P_\theta$, $\theta \in \Theta \subset \mathbb {R}$. Consider the estimation of $\theta$ under the loss $L(\left|\theta -a\right|)$, where $L$ is an increasing function on $[0,\infty )$. Let $\pi (\theta \mid x)$ be the posterior density (with respect to Lebesgue measure) of $\theta$ given $X=x$. Suppose that $\pi (\theta \mid x)$ is symmetric about $\delta (x)\in \Theta$ and that $\pi (\theta \mid x)$ is nondecreasing for $\theta \leq \delta (x)$ and nonincreasing for $\theta \geq \delta (x)$. Show that $\delta (x)$ is a Bayes action, assuming that all integrals involved are finite.

Let $X$ be a sample of size $1$ from the geometric distribution with mean $p^{-1}$, where $p\in (0,1]$. Consider the estimation of $p$ with the loss function $L(p,a)=(p-a)^2/p$.

Let $X$ be a sample from $P_\theta$ having probability density $f_\theta (x)=h(x)\mathrm{exp}\left(\theta^\tau x-\zeta (\theta )\right)$ with respect to $\nu$ on $\mathbb {R}^{p}$, where $\theta \in \mathbb {R}^{p}$. Let the prior be the Lebesgue measure on $\mathbb {R}^{p}$. Show that the generalized Bayes action under the loss $L(\theta ,a)=\left\| \mathbb {E}\left[X\right]-a\right\|^2$ is $\delta (x)=x$ when $X=x$ with $\int f_\theta (x)\, d\theta <\infty$.

Let $(X_1,\ldots ,X_n)$ be a random sample of random variables with the Lebesgue density $\sqrt{2/\pi }e^{-(x-\theta )^2/2}\; \mathbb {1}_{(\theta ,\infty )}(x)$, where $\theta \in \mathbb {R}$ is unknown. Find the generalized Bayes action for estimating $\theta$ under the squared error loss, when the (improper) prior of $\theta$ is the Lebesgue measure on $\mathbb {R}$.

Let $(X_1,\ldots ,X_n)$ be a random sample from $\mathscr {N}\left(\mu , \sigma^2\right)$ and $\pi (\mu ,\sigma^2)=\sigma^{-2}\; \mathbb {1}_{(0,\infty )}(\sigma^2)$ be an improper prior for $(\mu ,\sigma^2)$ with respect to the Lebesgue measure on $\mathbb {R}^{2}$.

In (ii)--(iv) of Exercise 4.1, assume that the parameters in priors are unknown. Using the method of moments, find empirical Bayes actions under the squared error loss.

Let $X=(X_1,\ldots ,X_n)$ be a random sample from $\mathscr {N}\left(\mu , \sigma^2\right)$ with an unknown $\mu \in \mathbb {R}$ and a known $\sigma^2>0$. Consider the prior $\Pi_{\mu \mid \xi }=\mathscr {N}\left(\mu_0, \sigma_0^2\right)$, $\xi =(\mu_0,\sigma_0^2)$, and the second-stage improper joint prior for $\xi$ be the product of $\mathscr {N}\left(a, v^2\right)$ and the Lebesgue measure on $(0,\infty )$, where $a$ and $v$ are known. Under the squared error loss, obtain a formula for the generalized Bayes action in terms of a one-dimensional integral.

Let $X=(X_1,\ldots ,X_n)$ be a random sample from the uniform distribution on $(0,\theta )$, where $\theta >0$ is unknown. Let $\pi (\theta )=ba^b\theta^{-(b+1)}\; \mathbb {1}_{(a,\infty )}(\theta )$ be a prior density with respect to the Lebesgue measure, where $b>1$ is known but $a>0$ is an unknown hyperparameter. Consider the estimation of $\theta$ under the squared error loss.

Let $(X_1,\ldots ,X_n)$ be a random sample from the exponential distribution on $(0,\infty )$ with scale parameter $1$. Suppose that we observe $T=X_1+\cdots +X_\theta$, where $\theta$ is an unknown positive integer. Consider the estimation of $\theta$ under the loss function $L(\theta ,a)=(\theta -a)^2/\theta$ and the geometric distribution with mean $p^{-1}$ as the prior for $\theta$, where $p\in (0,1)$ is known.

Let $(X_1,\ldots ,X_n)$ be a random sample of binary random variables with $\mathbb {P}\left(X_1=1\right)=p\in (0,1)$.

Let $X=(X_1,\ldots ,X_n)$ be a random sample from $\mathscr {N}\left(\mu , 1\right)$, $\mu \in \mathbb {R}$. Show that the sample mean $\overline{X}$ is an admissible estimator of $\mu$ under the loss function $L(\mu ,a)=\left|\mu -a\right|$.

Let $X$ be an observation from the negative binomial distribution with a known size $r$ and an unknown probability $p\in (0,1)$. Show that $(X+1)/(r+1)$ is an admissible estimator of $p^{-1}$ under the squared error loss.

Let $(X_1,\ldots ,X_n)$ be a random sample of binary random variables with $\mathbb {P}\left(X_1=1\right)=p\in (0,1)$.

Let $X=(X_1,\ldots ,X_n)$ be a random sample from the uniform distribution on $(\theta ,\theta +1)$, $\theta \in \mathbb {R}$. Consider the estimation of $\theta$ under the squared error loss. Let $\pi (\theta )$ be a continuous and positive Lebesgue density on $\mathbb {R}$. Derive the Bayes estimator under the prior $\pi$ and show that it is a consistent estimator of $\theta$.

Consider the linear model with observed vector $X$ having distribution $N_n(Z\beta ,\sigma^2I_n)$, where $Z$ is an $n\times p$ known matrix, $p<n$, $\beta \in \mathbb {R}^{p}$, and $\sigma^2>0$.

Let $(X_1,\ldots ,X_n)$ be a random sample of random variables with the Lebesgue density $\sqrt{2/\pi }e^{-(x-\theta )^2/2}\; \mathbb {1}_{(\theta ,\infty )}(x)$, where $\theta \in \mathbb {R}$ is unknown. Find the MRIE (minimum risk invariant estimator) of $\theta$ under the squared error loss.

Let $(X_1,\ldots ,X_n)$ be a random sample from the exponential distribution on $(\mu ,\infty )$ with a known scale parameter $\theta$, where $\mu \in \mathbb {R}$ is unknown. Let $X_{(1)}$ be the smallest order statistic. Show that

Let $X=(X_1,\ldots ,X_n)$ be a random sample from a population in a location family with unknown location parameter $\mu \in \mathbb {R}$ and $T$ be a location invariant estimator of $\mu$. Show that $T$ is an MRIE under the squared error loss if and only if $T$ is unbiased and $\mathbb {E}\left[T(X)U(X)\right]=0$ for any $U(X)$ satisfying $U(X_1+c,\ldots ,X_n+c)=U(X)$ for any $c$, $\operatorname {Var}\left[U\right]<\infty$, and $\mathbb {E}\left[U(X)\right]=0$ for any $\mu$.

Let $(X_1,\ldots ,X_n)$ be a random sample from the uniform distribution on $(0,\sigma )$ and consider the estimation of $\sigma >0$. Show that the MRIE of $\sigma$ is $2^{(n+1)^{-1}}X_{(n)}$ when the loss is $L(\sigma ,a)=\left|1-a/\sigma \right|$, where $X_{(n)}$ is the largest order statistic.

Let $(X_1,\ldots ,X_n)$ be a random sample from the Pareto distribution with Lebesgue density $\alpha \sigma^\alpha x^{-(\alpha +1)}\; \mathbb {1}_{(\sigma ,\infty )}(x)$, where $\sigma >0$ is an unknown parameter and $\alpha >2$ is known. Find the MRIE of $\sigma$ under the loss function $L(\sigma ,a)=(1-a/\sigma )^2$.

Let $(X_1,\ldots ,X_n)$ be a random sample from the exponential distribution on $(\mu ,\infty )$ with scale parameter $\sigma$, where $\mu \in \mathbb {R}$ and $\sigma >0$ are unknown.

Let $(X_1,\ldots ,X_n)$ be a random sample of binary random variables with $\mathbb {P}\left(X_1=1\right)=p\in (0,1)$. Let $T$ be a randomized estimator of $p$ with probability $n/(n+1)$ being the sample mean $\overline{X}$ and probability $1/(n+1)$ being $\frac12$. Under the squared error loss, show that $T$ has a constant risk that is smaller than the maximum risk of $\overline{X}$.

Let $X$ be a single sample from the geometric distribution with mean $p^{-1}$, where $p\in (0,1)$. Show that $\; \mathbb {1}_{\left\{ 1\right\} }(X)$ is a minimax estimator of $p$ under the loss function $L(p,a)=(a-p)^2/[p(1-p)]$.

Let $(X_1,\ldots ,X_m)$ be a random sample from $\mathscr {N}\left(\mu_x, \sigma_x^2\right)$ and $(Y_1,\ldots ,Y_n)$ be a random sample from $\mathscr {N}\left(\mu_y, \sigma_y^2\right)$. Assume that $X_i$'s and $Y_j$'s are independent. Consider the estimation of $\Delta =\mu_y-\mu_x$ under the squared error loss.

Consider the linear model with observed vector $X$ having distribution $N_n(Z\beta ,\sigma^2I_n)$, where $Z$ is an $n\times p$ known matrix, $p<n$, $\beta \in \mathbb {R}^{p}$, and $\sigma^2>0$, and the estimation of $l^\tau \beta$ under the squared error loss, where $l\in \mathcal R(Z)$. Show that the LSE $l^\tau \widehat{\beta }$ is minimax if $\sigma^2\in (0,c]$ with a constant $c$.

Let $X$ be an observation having the hypergeometric distribution with discrete probability density

where $N$ and $r$ are known and $\theta$ is an unknown integer between $1$ and $N$. Consider the estimation of $\theta /N$ under the squared error loss.

Let $X$ be an observation from $\mathscr {N}\left(\mu , 1\right)$ and let $\mu$ have the improper Lebesgue prior density $\pi (\mu )=e^\mu$. Under the squared error loss, show that the generalized Bayes estimator of $\mu$ is $X+1$, which is neither minimax nor admissible.

Let $X$ be an observation from the Poisson distribution with unknown mean $\theta >0$. Consider the estimation of $\theta$ under the squared error loss.

Let $(X_1,\ldots ,X_n)$ be a random sample from the uniform distribution on the interval $(\mu -\frac12,\mu +\frac12)$ with an unknown $\mu \in \mathbb {R}$. Under the squared error loss, show that $(X_{(1)}+X_{(n)})/2$ is the unique minimax estimator of $\mu$, where $X_{(j)}$ is the $j$th order statistic.

Let $(X_1,\ldots ,X_n)$ be a random sample from the exponential distribution on $(0,\infty )$ with unknown mean $\theta >0$ and $\overline{X}$ be the sample mean. Show that $(n\overline{X}+b)/(n+1)$ is an admissible estimator of $\theta$ under the squared error loss for any $b\geq 0$ and that $n\overline{X}/(n+1)$ is a minimax estimator of $\theta$ under the loss function $L(\theta ,a)=(a-\theta )^2/\theta^2$.

Let $X$ be a single observation. Consider the estimation of $\mathbb {E}\left[X\right]$ under the squared error loss.

Let $X$ be an observation from the distribution with Lebesgue density $\frac12c(\theta )e^{\theta x-\left|x\right|}$, $\left|\theta \right|<1$.

Let $X$ be an observation with the discrete probability density $f_\theta (x)=[x!(1-e^{-\theta })]^{-1}\theta^xe^{-\theta }\; \mathbb {1}_{\left\{ 1,2,\ldots \right\} }(x)$, where $\theta >0$ is unknown. Consider the estimation of $\theta /(1-e^{-\theta })$ under the squared error loss.

Suppose that $X$ is distributed as $N_p(\theta ,I_p)$, where $\theta \in \mathbb {R}^{p}$. Consider the estimation of $\theta$ under the loss $(a-\theta )^\tau Q(a-\theta )$ with a known positive definite $p\times p$ matrix $Q$. Show that the risk of the estimator

is equal to

Suppose that $X$ is distributed as $N_p(\theta ,\sigma^2D)$, where $\theta \in \mathbb {R}^{p}$ is unknown, $\sigma^2>0$ is unknown, and $D$ is a known $p\times p$ positive definite matrix. Consider the estimation of $\theta$ under the loss $\left\| a-\theta \right\|^2$. Show that the risk of the estimator

is equal to

Let $X=(X_1,\ldots ,X_n)$ be a random sample of random variables with probability density $f_\theta$. Find an MLE (maximum likelihood estimator) of $\theta$ in each of the following cases.