Unbiased Estimation

Let $X$ be a sample from $P\in \mathcal P$ and $\theta$ be a parameter. Show that if both $T_1(X)$ and $T_2(X)$ are UMVUE's (uniformly minimum variance unbiased estimators) of $\theta$ with finite variances, then $T_1(X)=T_2(X)$ a.s. $P$ for any $P\in \mathcal P$.

Let $(X_1,\ldots ,X_n)$ be a sample of binary random variables with $\mathbb {P}\left(X_i=1\right)=p\in (0,1)$.

Let $(X_1,\ldots ,X_n)$ be a random sample from $\mathscr {N}\left(\mu , \sigma^2\right)$ with an unknown $\mu \in \mathbb {R}$ and a known $\sigma^2>0$.

Let $(X_1,\ldots ,X_m)$ be a random sample from $\mathscr {N}\left(\mu_x, \sigma_x^2\right)$ and let $Y_1,\ldots ,Y_n$ be a random sample from $\mathscr {N}\left(\mu_y, \sigma_y^2\right)$. Assume that $X_i$'s and $Y_j$'s are independent.

Let $(X_1,\ldots ,X_n)$, $n>2$, be a random sample from the uniform distribution on the interval $(\theta_1-\theta_2,\theta_1+\theta_2)$, where $\theta_1\in \mathbb {R}$ and $\theta_2>0$. Find the UMVUE's of $\theta_j$, $j=1,2$, and $\theta_1/\theta_2$.

Let $(X_1,\ldots ,X_n)$ be a random sample from the exponential distribution on $(a,\infty )$ with scale parameter $\theta$, where $\theta >0$ and $a\in \mathbb {R}$.

Let $(X_1,\ldots ,X_n)$ be a random sample from the Pareto distribution with Lebesgue density $\theta a^\theta x^{-(\theta +1)}\; \mathbb {1}_{(a,\infty )}(x)$, where $\theta >0$ and $a>0$.

Let $X$ be a random variable having the negative binomial distribution with

where $p\in (0,1)$ and $r$ is a known positive integer.

Let $(X_1,\ldots ,X_n)$ be a random sample from the Poisson distribution truncated at $0$, i.e., $\mathbb {P}\left(X_i=x\right)=(e^\theta -1)^{-1}\theta^x/x!$, $x=1,2,\ldots$, $\theta >0$. Find the UMVUE of $\theta$ when $n=1,2$.

Let $X_1,\ldots ,X_n$ be a random sample from the log-distribution with

$p\in (0,1)$. Let $k$ be a fixed positive integer.

Let $Y_1,\ldots ,Y_n$ be a random sample from the uniform distribution on the interval $(0,\theta )$ with an unknown $\theta \in (1,\infty )$.

Let $(X_1,\ldots ,X_n)$ be a random sample from $P\in \mathcal P$ containing all symmetric distributions with finite means and with Lebesgue densities on $\mathbb {R}$.

Suppose that $T$ is a UMVUE of an unknown parameter $\theta$. Show that $T^k$ is a UMVUE of $\mathbb {E}\left[T^k\right]$, where $k$ is any positive integer for which $\mathbb {E}\left[T^{2k}\right]<\infty$.

Let $X$ be a random variable having the Lebesgue density $[(1-\theta )+\theta /(2\sqrt x)]\; \mathbb {1}_{(0,1)}(x)$, where $\theta \in [0,1]$. Show that there is no UMVUE of $\theta$ based on an observation $X$.

Let $X$ be a random sample with $\mathbb {P}\left(X=-1\right)=2p(1-p)$ and $\mathbb {P}\left(X=k\right)=p^k(1-p)^{3-k}$, $k=0,1,2,3$, where $p\in (0,1)$.

Let $(X_1,\ldots ,X_n)$ be a random sample from the exponential distribution with density $\theta^{-1}e^{-(x-a)/\theta }\; \mathbb {1}_{(a,\infty )}(x)$, where $a\leq 0$ and $\theta$ is known. Obtain a UMVUE of $a$.

Let $(X_1,\ldots ,X_n)$ be a random sample from the distribution on $\mathbb {R}$ with Lebesgue density $\theta a^\theta x^{-(\theta +1)}\; \mathbb {1}_{(a,\infty )}(x)$, where $a\in (0,1]$ and $\theta$ is known. Obtain a UMVUE of $a$.

Let $(X_1,\ldots ,X_n)$ be a random sample from the population in a family $\mathcal P$ as described in Exercise 2.18 of Chapter 2. Find a UMVUE of $\theta$.

Find a function of $\theta$ for which the amount of information is independent of $\theta$, when $P_\theta$ is

1. the Poisson distribution with unknown mean $\theta >0$;

2. the binomial distribution with known size $r$ and unknown probability $\theta \in (0,1)$;

3. the gamma distribution with known shape parameter $\alpha$ and unknown scale parameter $\theta >0$.

Let $(X_1,\ldots ,X_n)$ be a random sample from a distribution on $\mathbb {R}$ with the Lebesgue density $\frac1\sigma f\left(\frac{x-\mu }\sigma \right)$, where $f(x)>0$ is a known Lebesgue density and $f'(x)$ exists for all $x\in \mathbb {R}$, $\mu \in \mathbb {R}$, and $\sigma >0$. Let $\theta =(\mu ,\sigma )$. Show that the Fisher information about $\theta$ contained in $X_1,\ldots ,X_n$ is

assuming that all integrals are finite.

Let $X$ be a sample having a probability density $f_\theta (x)$ with respect to $\nu$, where $\theta$ is a $k$-vector of unknown parameters. Let $T(X)$ be a statistic having a probability density $g_\theta (t)$ with respect to $\lambda$. Suppose that $\frac\partial {\partial \theta }f_\theta (x)$ and $\frac\partial {\partial \theta }g_\theta (t)$ exist for any $x$ and $t$ and that, on any set $\left\{ \left\| \theta \right\| \leq c\right\}$, there are functions $u_c(x)$ and $v_c(t)$ such that $\left|\frac\partial {\partial \theta }f_\theta (x)\right|\leq u_c(x)$, $\left|\frac\partial {\partial \theta }g_\theta (t)\right|\leq v_c(t)$, $\int u_c(x)\, d\nu <\infty$, and $\int v_c(t)\, d\lambda <\infty$. Show that

Let $(X_1,\ldots ,X_n)$ be a random sample from the uniform distribution on the interval $(0,\theta )$ with $\theta >0$.

Let $X$ be an observation with Lebesgue density $(2\theta )^{-1}e^{-\left|x\right|/\theta }$ with unknown $\theta >0$. Find the UMVUE's of the parameters $\theta$, $\theta^r$ ($r>1$), and $(1+\theta )^{-1}$ and, in each case, determine whether the variance of the UMVUE attains the Cramer--Rao lower bound.

Let $(X_1,\ldots ,X_n)$ be a random sample from $\mathscr {N}\left(\mu , \sigma^2\right)$ with an unknown $\mu \in \mathbb {R}$ and a known $\sigma^2>0$. Find the UMVUE of $e^{t\mu }$ with a fixed $t\ne 0$ and show that the variance of the UMVUE is larger than the Cramer--Rao lower bound but the ratio of the variance of the UMVUE over the Cramer--Rao lower bound converges to $1$ as $n\to \infty$.

Let $X_1,X_2,\ldots$ be independent and identically distributed random variables, $m$ be a positive integer, and $h(x_1,\ldots ,x_m)$ be a function on $\mathbb {R}^{m}$ such that $\mathbb {E}\left[[h(X_1,\ldots ,X_m)]^2\right]<\infty$ and $h$ is symmetric in its $m$ arguments. A U-statistic with kernel $h$ (of order $m$) is defined as

where $\sum_{1\leq i_1<\cdots <i_m\leq n}$ denotes the summation over the $\binom nm$ combinations of $m$ distinct elements $\left\{ i_1,\ldots ,i_m\right\}$ from $\left\{ 1,\ldots ,n\right\}$. For $k=1,\ldots ,m$, define $h_k(x_1,\ldots ,x_k)=\mathbb {E}\left[h(x_1,\ldots ,x_k,X_{k+1},\ldots ,X_m)\right]$ and $\zeta_k=\operatorname {Var}\left[h_k(X_1,\ldots ,X_k)\right]$. Show that

Let $h(x_1,x_2,x_3)=\; \mathbb {1}_{(-\infty ,0)}(x_1+x_2+x_3)$. Find $h_k$ and $\zeta_k$, $k=1,2,3$, for the U-statistic with kernel $h$ based on independent random variables $X_1,X_2,\ldots$ with a common cumulative distribution function $F$.

Let $X_1,\ldots ,X_n$ be a random sample of random variables having finite $\mathbb {E}\left[X_1^2\right]$ and $\mathbb {E}\left[X_1^{-2}\right]$. Let $\mu =\mathbb {E}\left[X_1\right]$ and $\bar\mu =\mathbb {E}\left[X_1^{-1}\right]$. Find a U-statistic that is an unbiased estimator of $\mu \bar\mu$ and derive its variance and asymptotic distribution.

Suppose that

where $\alpha_i$ and $\theta$ are unknown parameters, $t_{ij}$ are known constants, and $\varepsilon_{ij}$ are independent and identically distributed random variables with mean $0$. Find explicit forms for the least squares estimators (LSE's) of $\theta$, $\alpha_i$, $i=1,\ldots ,a$.

Consider the polynomial model

where $\varepsilon_i$'s are independent and identically distributed random variables with mean $0$. Suppose that $n=12$, $t_i=-1$, $i=1,\ldots ,4$, $t_i=0$, $i=5,\ldots ,8$, and $t_i=1$, $i=9,\ldots ,12$. Show whether the following parameters are estimable (i.e., they can be unbiasedly estimated): $\beta_0+\beta_2$, $\beta_1$, $\beta_0-\beta_1$, $\beta_1+\beta_3$, and $\beta_0+\beta_1+\beta_2+\beta_3$.

Consider the one-way ANOVA model

where $\mu$ and $\alpha_i$ are unknown parameters and $\varepsilon_{ij}$ are independent and identically distributed random variables with mean $0$. Let

and $\beta =(\mu ,\alpha_1,\ldots ,\alpha_m)$. Find the matrix $Z$ in the linear model $X=Z\beta +\varepsilon$, the matrix $Z^\tau Z$, and the form of $l$ for estimable $l^\tau \beta$.

Consider the two-way balanced ANOVA model

where $a$, $b$, and $c$ are some positive integers, $\varepsilon_{ijk}$'s are independent and identically distributed random variables with mean $0$, and $\mu$, $\alpha_i$'s, $\beta_j$'s, and $\gamma_{ij}$'s are unknown parameters. Let $X$ be the vector of $X_{ijk}$'s, $\varepsilon$ be the vector of $\varepsilon_{ijk}$'s, and $\beta =(\mu ,\alpha_1,\ldots ,\alpha_a,\beta_1,\ldots ,\beta_b,\gamma_{11},\ldots ,\gamma_{1b},\ldots ,\gamma_{a1},\ldots ,\gamma_{ab})$.

Assume that $X$ is a random $n$-vector from the multivariate normal distribution $N_n(Z\beta ,\sigma^2I_n)$, where $Z$ is an $n\times p$ known matrix of rank $r\leq p<n$, $\beta$ is a $p$-vector of unknown parameters, $I_n$ is the identity matrix of order $n$, and $\sigma^2>0$ is unknown. Find the UMVUE's of $(l^\tau \beta )^2$, $l^\tau \beta /\sigma$, and $(l^\tau \beta /\sigma )^2$ for an estimable $l^\tau \beta$.

Consider the one-way random effects model

where $\mu \in \mathbb {R}$ is an unknown parameter, $A_i$'s are independent and identically distributed as $\mathscr {N}\left(0, \sigma_a^2\right)$, $e_{ij}$'s are independent and identically distributed as $\mathscr {N}\left(0, \sigma^2\right)$, and $A_i$'s and $e_{ij}$'s are independent. Based on observed $X_{ij}$'s, show that the family of populations is an exponential family with sufficient and complete statistics $\overline{X}_{\cdot \cdot }$, $S_A=n\sum_{i=1}^m(\overline{X}_{i\cdot }-\overline{X}_{\cdot \cdot })^2$, and $S_E=\sum_{i=1}^m\sum_{j=1}^n(X_{ij}-\overline{X}_{i\cdot })^2$, where $\overline{X}_{\cdot \cdot }=(nm)^{-1}\sum_{i=1}^m\sum_{j=1}^nX_{ij}$ and $\overline{X}_{i\cdot }=n^{-1}\sum_{j=1}^nX_{ij}$. Find the UMVUE's of $\mu$, $\sigma_a^2$, and $\sigma^2$.

Consider the linear model $X=Z\beta +\varepsilon$, where $Z$ is a known $n\times p$ matrix, $\beta$ is a $p$-vector of unknown parameters, and $\varepsilon$ is a random $n$-vector whose components are independent and identically distributed with mean $0$ and Lebesgue density $\sigma^{-1}f(x/\sigma )$, where $f$ is a known Lebesgue density and $\sigma >0$ is unknown. Find the Fisher information about $(\beta ,\sigma )$ contained in $X$.

Consider the linear model $X=Z\beta +\varepsilon$, where $Z$ is a known $n\times p$ matrix, $\beta$ is a $p$-vector of unknown parameters, and $\varepsilon$ is a random $n$-vector whose components are independent and identically distributed with mean $0$ and variance $\sigma^2$. Let $c\in \mathbb {R}^{p}$. Show that if the equation $c=Z^\tau y$ has a solution, then there is a unique solution $y_0\in \mathcal R(Z^\tau )$ such that $\operatorname {Var}\left[y_0^\tau X\right]\leq \operatorname {Var}\left[y^\tau X\right]$ for any other solution of $c=Z^\tau y$.

Consider the linear model $X=Z\beta +\varepsilon$, where $Z$ is a known $n\times p$ matrix, $\beta$ is a $p$-vector of unknown parameters, and $\varepsilon$ is a random $n$-vector whose components are independent and identically distributed with mean $0$ and variance $\sigma^2$. Let $X_i$ be the $i$th component of $X$, $Z_i$ be the $i$th row of $Z$, $h_{ij}$ be the $(i,j)$th element of $Z(Z^\tau Z)^{-}Z^\tau$, $h_i=h_{ii}$, $\widehat{\beta }$ be an LSE of $\beta$, and $\widehat{X}_i=Z_i^\tau \widehat{\beta }$. Show that

Consider the linear model $X=Z\beta +\varepsilon$, where $Z$ is a known $n\times p$ matrix, $\beta$ is a $p$-vector of unknown parameters, and $\varepsilon$ is a random $n$-vector whose components are independent and identically distributed with mean $0$ and variance $\sigma^2$. Let $Z=(Z_1,Z_2)$ and $\beta =(\beta_1,\beta_2)$, where $Z_j$ is $n\times p_j$ and $\beta_j$ is a $p_j$-vector, $j=1,2$. Assume that $(Z_1^\tau Z_1)^{-1}$ and $[Z_2^\tau Z_2-Z_2^\tau Z_1(Z_1^\tau Z_1)^{-1}Z_1^\tau Z_2]^{-1}$ exist.

Consider the linear model $X=Z\beta +\varepsilon$, where $Z$ is a known $n\times p$ matrix, $\beta$ is a $p$-vector of unknown parameters, and $\varepsilon$ is a random $n$-vector with $\mathbb {E}\left[\varepsilon \right]=0$ and finite $\operatorname {Var}\left[\varepsilon \right]=\Sigma$. Show the following statements are equivalent: