The Information Inequality

Under the assumptions of Problem 1.3, determine for each $p_{1}$, the value $L_{V}\left(p_{1}\right)$ of the LMVU estimator of $p$ at $p_{1}$ and compare the function $L_{V}(p), 0<p<1$ with the variance $V_{p_{0}}(p)$ of the estimator which is LMVU at

Determine the conditions under which equality holds in (5.1).

Verify $I(\theta )$ for the distributions of Table 5.1.

If $X$ is normal with mean zero and standard deviation $\sigma$, determine $I(\sigma )$.

Find $I(p)$ for the negative binomial distribution.

If $X$ is distributed as $P(\lambda )$, show that the information it contains about $\sqrt{\lambda }$ is independent of $\lambda$.

Verify the following statements, asserted by Basu (1988, Chapter 1), which illustrate the relationship between information, sufficiency, and ancillarity. Suppose that we let $I(\theta )=\mathbb {E}_{\theta }\left[-\partial^{2} / \partial \theta^{2} \log f(x \mid \theta )\right]$ be the information in $X$ about $\theta$ and let $J(\theta )=\mathbb {E}_{\theta }\left[-\partial^{2} / \partial \theta^{2} \log g(T \mid \theta )\right]$ be the information about $\theta$ contained in a statistic $T$, where $g(\cdot \mid \theta )$ is the density function of $T$. Define $\lambda (\theta )=I(\theta )-J(\theta )$, a measure of information lost by using $T$ instead of $X$. Under suitable regularity conditions, show that

Find a function of $\theta$ for which the amount of information is independent of $\theta$ :

For inverse binomial sampling (see Example 3.2):

Show that (5.13) can be differentiated by differentiating under the integral sign when $p_{\theta }(x)$ is given by (5.24), for each of the distributions of Table 5.2.

Verify the entries of Table 5.2.

Evaluate (5.25) when $f$ is the density of Student's $t$-distribution with $v$ degrees of freedom.

For the distribution with density (5.24), show that $I(\theta )$ is independent of $\theta$.

Verify

For the location $t$ density, calculate the information inequality bound for unbiased estimators of $\theta$.

If $p_{\theta }(x)$ is given by 1.5.1 with $s=1$ and $T(x)=\delta (x)$, show that $\operatorname {var}[\delta (X)]$ attains the lower bound (5.31) and is the only estimator to do so.

Show that if a given function $g(\theta )$ has an unbiased estimator, there exists an unbiased estimator $\delta$ which for all $\theta$ values attains the lower bound (5.1) for some $\psi (x, \theta )$ satisfying (5.2) if and only if $g(\theta )$ has a UMVU estimator $\delta_{0}$. (Blyth 1974).

Show that if $\mathbb {E}_{\theta }\left[\delta \right]=g(\theta )$, and $\operatorname {var}(\delta )$ attains the information inequality bound (5.31), then

If $\mathbb {E}_{\theta }\left[\delta \right]=g(\theta )$, the information inequality lower bound is $I B(\theta )=\left[g^{\prime }(\theta )\right]^{2} / I(\theta )$. If $\theta =h(\xi )$ where $h$ is differentiable, show that $I B(\xi )=I B(\theta )$.

Let $X$ be an observation from the normal mixture density

where $\Omega$ is any neighborhood of zero. Thus, the random variable $X$ is either $N(\theta , 1)$ or $N(-\theta , 1)$, each with probability $1 / 2$. Show that $\theta =0$ is a singular point, that is, if there exists an unbiased estimator of $\theta$ it will have infinite variance at $\theta =0$.

Let $X_{1}, \ldots , X_{n}$ be a sample from the Poisson $(\lambda )$ distribution truncated on the left at 0, i.e., with sample space $\mathcal{X}=\left\{ 1,2,3, \ldots \right\}$ (see Problem 3.20). Show that the Cramér-Rao lower bound for the variance of unbiased estimators of $\lambda$ is

and is not attained by the UMVU estimator. (It is, however, the asymptotic variance of the ML estimator.)

Let $X_{1}, \ldots , X_{n}$ be iid according to a density $p(x, \theta )$ which is positive for all $x$. Then, the variance of any unbiased estimator $\delta$ of $\theta$ satisfies

If $X_{1}, \ldots , X_{n}$ are iid as $N\left(\theta , \sigma^{2}\right)$ where $\sigma$ is known and $\theta$ is known to have one of the values 0, ±1, ±2, . . ., the inequality of the preceding problem shows that any unbiased estimator $\delta$ of the restricted parameter $\theta$ satisfies

where $\Delta =\theta -\theta_{0}$, and hence $\sup_{\Delta \neq 0} \operatorname {var}_{\theta_{0}}(\delta ) \geq 1 /\left[e^{n / \sigma^{2}}-1\right]$.

Under the assumptions of the preceding problem, let $\bar{X}^{*}$ be the integer closest to $\bar{X}$.

Kiefer inequality.

Verify directly that the following families of densities satisfy (5.38).

Extend condition (5.38) to vector-valued parameters, and show that it is satisfied by the exponential family (1.5.1) for $s>1$.

Show that the assumption (5.36(b)) implies (5.38), so Theorem 5.15 is, in fact, a corollary of Theorem 5.10.

Show that (5.38) is satisfied if either of the following is true:

Brown and Gajek (1990) give two different sufficient conditions for (8.2) to hold, which are given below. Show that each implies (8.2). (Note that, in the progression from (a) to (b) the conditions become weaker, thus more widely applicable and harder to check.)

Let $\mathcal{F}$ be the class of all unimodal symmetric densities or, more generally, densities symmetric around zero and satisfying $f(x) \leq f(0)$ for all $x$. Show that

and that the minimum is attained by the uniform $\left(-\frac{1}{2}, \frac{1}{2}\right)$ distribution. Thus, the uniform distribution has minimum variance among symmetric unimodal distributions. (See Example 4.8.6 for large-sample properties of the scale uniform.)