Linear Regression: Estimation and Distribution Theory

Let $Y_{i} = a_{i} \beta_{1}+b_{i} \beta_{2}+\varepsilon_{i}(i = 1,2, \ldots , n)$, where the $a_{i}, b_{i}$ are known and the $\varepsilon_{i}$ are independently and identically distributed as $N\left(0, \sigma^{2}\right)$. Find a necessary and sufficient condition for the least squares estimates of $\beta_{1}$ and $\beta_{2}$ to be independent.

Let $\mathbf{Y} = \mathbf{\theta }+\mathbf{\varepsilon }$, where $\mathbb {E}\left[\varepsilon \right] = 0$. Prove that the value of $\mathbf{\theta }$ that minimizes $\left\| \mathbf{Y}-\mathbf{\theta }\right\|^{2}$ subject to $\mathbf{A} \mathbf{\theta } = \mathbf{0}$, where $\mathbf{A}$ is a known $q \times n$ matrix of rank $q$, is

Let $\mathbf{Y} = \mathbf{X} \mathbf{\beta }+\varepsilon$, where $\mathbb {E}\left[\varepsilon \right] = \mathbf{0}, \operatorname {Var}\left[\varepsilon \right] = \sigma^{2} \mathbf{I}_{n}$, and $\mathbf{X}$ is $n \times p$ of rank $p$. If $\mathbf{X}$ and $\mathbf{\beta }$ are partitioned in the form

prove that the least squares estimate $\widehat{\mathbf{\beta }}_{2}$ of $\mathbf{\beta }_{\mathbf{2}}$ is given by

Find $\operatorname {Var}\left[\widehat{\mathbf{\beta }}_{2}\right]$.

Suppose that $\mathbb {E}\left[\mathbf{Y}\right] = \mathbf{X} \mathbf{\beta }$ and $\operatorname {Var}\left[\mathbf{Y}\right] = \sigma^{2} \mathbf{I}_{n}$. Prove that $\mathbf{a}^{T} \mathbf{Y}$ is the linear unbiased estimate of $\mathbb {E}\left[\mathbf{a}^{T} \mathbf{Y}\right]$ with minimum variance if and only if $\operatorname {Cov}\left[ \mathbf{a}^{T} \mathbf{Y}, \mathbf{b}^{T} \mathbf{Y} \right] = 0$ for all $\mathbf{b}$ such that $\mathbb {E}\left[\mathbf{b}^{T} \mathbf{Y}\right] = 0$ (i.e., $\mathbf{b}^{T} \mathbf{X} = \mathbf{0}^{T}$).

If $\mathbf{X}$ has full rank and $\widehat{\mathbf{Y}} = \mathbf{X} \widehat{\mathbf{\beta }}$, prove that

Estimate the weights $\beta_{i}(i = 1,2,3,4)$ of four objects from the following weighing data (see Exercises 3e, No. 5, at the end of Section 3.6 for notation):

Three parcels are weighed at a post office singly, in pairs, and all together, giving weights $Y_{i j k}(i, j, k = 0,1)$, the suffix 1 denoting the presence of a particular parcel and the suffix 0 denoting its absence. Find the least squares estimates of the weights.

An experimenter wishes to estimate the density $d$ of a liquid by weighing known volumes of the liquid. Let $Y_{i}$ be the weight for volume $x_{i}(i = 1,2, \ldots , n)$ and let $\mathbb {E}\left[Y_{i}\right] = d x_{i}$ and $\operatorname {Var}\left[Y_{i}\right] = \sigma^{2} f\left(x_{i}\right)$. Find the least squares estimate of $d$ for the following cases:

Let $Y_{i} = \beta_{0}+\beta_{1} x_{i}+\varepsilon_{i}(i = 1,2,3)$, where $\mathbb {E}\left[\varepsilon \right] = 0, \operatorname {Var}\left[\varepsilon \right] = \sigma^{2} \mathrm{~ V}$ with

and $x_{1} = -1, x_{2} = 0$, and $x_{3} = 1$. Show that the generalized least squares estimates of $\beta_{0}$ and $\beta_{1}$ are

where $r = 1+\rho +2 a^{2}-4 a \rho$. Also prove the following:

If $\mathbf{X}$ is not of full rank, show that any solution $\mathbf{\beta }$ of $\mathbf{X}^{T} \mathbf{V}^{-1} \mathbf{X} \mathbf{\beta } = \mathbf{X}^{T} \mathbf{V}^{-1} \mathbf{Y}$ minimizes $(\mathbf{Y}-\mathbf{X} \mathbf{\beta })^{T} \mathbf{V}^{-1}(\mathbf{Y}-\mathbf{X} \mathbf{\beta })$.

Let

and

where $\mathbb {E}\left[\varepsilon_{i}\right] = 0(i = 1,2,3)$. Find the least squares estimates of $\partial_{1}$ and $\theta_{2}$. If the equations above are augmented to

find the least squares estimate of $\theta_{3}$.

Given the usual full-rank regression model, prove that the random variables $\bar{Y}$ and $\sum_{i}\left(Y_{i}-\widehat{Y}_{i}\right)^{2}$ are statistically independent.

Let $Y_{i} = \beta x_{i}+u_{i}, x_{i} > 0(i = 1,2, \ldots , n)$, where $u_{i} = \rho u_{i-1}+\varepsilon_{i}$ and the $\varepsilon_{i}$ are independently distributed as $N\left(0, \sigma^{2}\right)$. If $\widehat{\beta }$ is the ordinary least squares estimate of $\beta$, prove that $\operatorname {Var}\left[\widehat{\beta }\right]$ is inflated when $\rho > 0$.

Suppose that $\mathbb {E}\left[Y_{t}\right] = \beta_{0}+\beta_{1} \cos \left(2 \pi k_{1} t / n\right)+\beta_{2} \sin \left(2 \pi k_{2} t / n\right)$, where $t = 1,2, \ldots , n$, and $k_{1}$ and $k_{2}$ are positive integers. Find the least squares estimates of $\beta_{0}, \beta_{1}$, and $\beta_{2}$.

Suppose that $\mathbb {E}\left[Y_{i}\right] = \alpha_{0}+\beta_{1}\left(x_{i 1}-\bar{x}_{1}\right)+\beta_{2}\left(x_{i 2}-\bar{x}_{2}\right), i = 1,2, \ldots , n$. Show that the least squares estimates of $\alpha_{0}, \beta_{1}$, and $\beta_{2}$ can be obtained by the following two-stage procedure: (i) Fit the model $\mathbb {E}\left[Y_{i}\right] = \alpha_{0}+\beta_{1}\left(x_{i 1}-\bar{x}_{1}\right)$. (ii) Regress the residuals from (i) on ($x_{i 2}-\bar{x}_{2}$).

Show that if $\mathbf{X}$ has full rank,

and hence deduce that the left side is minimized uniquely when $\mathbf{\beta } = \widehat{\mathbf{\beta }}$.

If $\mathbf{X}$ has full rank, prove that $\sum_{i = 1}^{n}\left(Y_{i}-\widehat{Y}_{i}\right) = 0$. Hint: Consider the first column of $\mathbf{X}$.

Let

where $\mathbb {E}\left[\varepsilon_{i}\right] = 0(i = 1,2,3)$. Find the least squares estimates of $\theta$ and $\phi$.

Consider the regression model

where $x_{1} = -1, x_{2} = 0$, and $x_{3} = +1$. Find the least squares estimates of $\beta_{0}, \beta_{1}$, and $\beta_{2}$. Show that the least squares estimates of $\beta_{0}$ and $\beta_{1}$ are unchanged if $\beta_{2} = 0$.

The tension $T$ observed in a nonextensible string required to maintain a body of unknown weight $w$ in equilibrium on a smooth inclined plane of angle $\theta (0 < \theta < \pi / 2)$ is a random variable with mean $\mathbb {E}\left[T\right] = w \sin \theta$. If for $\theta = \theta_{i}(i = 1,2, \ldots , n)$ the corresponding values of $T$ are $T_{i}$ ($i = 1,2, \ldots , n$), find the least squares estimate of $w$.

If $\mathbf{X}$ has full rank, so that $\mathbf{P} = \mathbf{X}\left(\mathbf{X}^{T} \mathbf{X}\right)^{-1} \mathbf{X}^{T}$, prove that $\mathcal{C}(\mathbf{P}) = \mathcal{C}(\mathbf{X})$.

For a general regression model in which $\mathbf{X}$ may or may not have full rank, show that

Suppose that we scale the explanatory variables so that $x_{i j} = k_{j} w_{i j}$ for all $i, j$. By expressing $\mathbf{X}$ in terms of a new matrix $\mathbf{W}$, prove that $\widehat{\mathbf{Y}}$ remains unchanged under this change of scale.

Let $Y_{i} = \beta_{0}+\beta_{1} x_{i}+\varepsilon_{i}(i = 1,2, \ldots , n)$, where $\mathbb {E}\left[\varepsilon \right] = 0$ and $\operatorname {Var}\left[\varepsilon \right] = \sigma^{2} \mathbf{I}_{n}$. Find the least squares estimates of $\beta_{0}$ and $\beta_{1}$. Prove that they are uncorrelated if and only if $\bar{x} = 0$.

In order to estimate two parameters $\theta$ and $\phi$ it is possible to make observations of three types: (a) the first type have expectation $\theta$, (b) the second type have expectation $\theta +\phi$, and (c) the third type have expectation $\theta -2 \phi$. All observations are subject to uncorrelated errors of mean zero and constant variance. If $m$ observations of type (a), $m$ observations of (b), and $n$ observations of type (c) are made, find the least squares estimates $\widehat{\theta }$ and $\widehat{\phi }$. Prove that these estimates are uncorrelated if $m = 2 n$.

Let $Y_{1}, Y_{2}, \ldots , Y_{n}$ be a random sample from $N\left(\theta , \sigma^{2}\right)$. Find the linear unbiased estimate of $\theta$ with minimum variance.

Let

where $\bar{x}_{j} = \sum_{i = 1}^{n} x_{i j} / n, \mathbb {E}\left[\varepsilon \right] = \mathbf{0}$, and $\operatorname {Var}\left[\varepsilon \right] = \sigma^{2} \mathbf{I}_{n}$. If $\widehat{\beta }_{1}$ is the least squares estimate of $\beta_{1}$, show that

where $r$ is the correlation coefficient of the $n$ pairs $\left(x_{i 1}, x_{i 2}\right)$.

Suppose that $\mathbf{Y} \sim N_{n}\left(\mathbf{X} \mathbf{\beta }, \sigma^{2} \mathbf{I}_{n}\right)$, where $\mathbf{X}$ is $n \times p$ of $\operatorname {rank} p$.

Let $Y_{1}, Y_{2}, \ldots , Y_{n}$ be independently and identically distributed with mean $\theta$ and variance $\sigma^{2}$. Find the nonnegative quadratic unbiased estimate of $\sigma^{2}$ with the minimum variance.

Given $Y_{1}, Y_{2}, \ldots , Y_{n}$ independently distributed as $N\left(\theta , \sigma^{2}\right)$, use Theorem 3.5 to prove that:

Use Theorem 2.5 to prove that for the full-rank regression model, RSS is independent of $(\widehat{\mathbf{\beta }}-\mathbf{\beta })^{T} \mathbf{X}^{T} \mathbf{X}(\widehat{\mathbf{\beta }}-\mathbf{\beta })$.

Prove the statement above that the minimum is given by $\lambda_{j} = c(j = 1,2, \ldots , p-1$).

It is required to fit a regression model of the form

where $\phi (x)$ is a second-degree polynomial. If $x_{1} = -1, x_{2} = 0$, and $x_{3} = 1$, find $\phi$ such that the design matrix $\mathbf{X}$ has mutually orthogonal columns.

Suppose that $\mathbf{X} = \left(\mathbf{x}^{(0)}, \mathbf{x}^{(1)}, \ldots , \mathbf{x}^{(p-1)}, \mathbf{x}^{(p)}\right) = \left(\mathbf{W}, \mathbf{x}^{(p)}\right)$ has linearly independent columns.

What modifications in the statement of Example 3.3 proved above can be made if the term $\beta_{0}$ is omitted?

Suppose that we wish to find the weights $\beta_{i}(i = 1,2, \ldots , k)$ of $k$ objects. One method is to weigh each object $r$ times and take the average; this requires a total of $k r$ weighings, and the variance of each average is $\sigma^{2} / r$ ($\sigma^{2}$ being the variance of the weighing error). Another method is to weigh the objects in combinations; some of the objects are distributed between the two pans and weights are placed in one pan to achieve equilibrium. The regression model for such a scheme is

where $x_{i} = 0,1$, or -1 according as the $i$ th object is not used, placed in the left pan or in the right pan, $\varepsilon$ is the weighing error (assumed to be the same for all weighings), and $Y$ is the weight required for equilibrium ($Y$ is regarded as negative if placed in the left pan). After $n$ such weighing operations we can find the least squares estimates $\widehat{\beta }_{i}$ of the weights.

Prove that

Prove that $\widehat{\gamma }_{G}$ can be obtained by replacing $\mathbf{Y}$ by $\mathbf{Y}-\mathbf{Z} \gamma$ in $\mathbf{Y}^{T} \mathbf{R Y}$ and minimizing with respect to $\gamma$. Show further that the minimum value thus obtained is $\mathbf{Y}^{T} \mathbf{R}_{G} \mathbf{Y}$.

If $\widehat{\mathbf{\beta }}_{G} = \left(\widehat{\beta }_{G, j}\right)$ and $\widehat{\mathbf{\beta }} = \left(\widehat{\beta }_{j}\right)$, use Theorem 3.6 (iv) to prove that

Given that $Y_{1}, Y_{2}, \ldots , Y_{n}$ are independently distributed as $N\left(\theta , \sigma^{2}\right)$, find the least squares estimate of $\theta$.

By considering the identity $\mathbf{Y}-\widehat{\mathbf{Y}}_{H} = \mathbf{Y}-\widehat{\mathbf{Y}}+\widehat{\mathbf{Y}}-\widehat{\mathbf{Y}}_{H}$, prove that

Prove that

Hence deduce that

where $\widehat{\beta }_{H j}$ and $\widehat{\beta }_{j}$ are the $j$ th elements of $\widehat{\mathbf{\beta }}_{H}$ and $\widehat{\mathbf{\beta }}$, respectively.

Show that

If $\mathbf{X}$ is $n \times p$ of $\operatorname {rank} p$ and $\mathbf{B}$ is $p \times q$ of $\operatorname {rank} q$, show that $\operatorname {rank}(\mathbf{X B}) = q$.

Suppose that $\mathbf{X}$ does not have full rank, and let $\widehat{\mathbf{\beta }}_{i}(i = 1,2)$ be any two solutions of the normal equations. Show directly that

If the columns of $\mathbf{X}$ are linearly dependent, prove that there is no matrix $\mathbf{C}$ such that $\mathbf{C Y}$ is an unbiased estimate of $\mathbf{\beta }$.

Prove that $\mathbf{a}^{T} \mathbb {E}\left[\widehat{\mathbf{\beta }}\right]$ is an estimable function of $\mathbf{\beta }$.

If $\mathbf{a}_{1}^{T} \beta , \mathbf{a}_{2}^{T} \beta , \ldots , \mathbf{a}_{k}^{T} \beta$ are estimable, prove that any linear combination of these is also estimable.

If $\mathbf{a}^{T} \widehat{\mathbf{\beta }}$ is invariant with respect to $\widehat{\mathbf{\beta }}$, prove that $\mathbf{a}^{T} \mathbf{\beta }$ is estimable.

Prove that $\mathbf{a}^{T} \mathbf{\beta }$ is estimable if and only if

If $\mathbf{a}^{T} \mathbf{\beta }$ is an estimable function, prove that

Prove that all linear functions $\mathbf{a}^{T} \mathbf{\beta }$ are estimable if and only if the columns of $\mathbf{X}$ are linearly independent.

If $\mathbf{P}$ projects onto $\mathcal{C}(\mathbf{X})$, show that $\mathbf{Z}^{T}\left(\mathbf{I}_{n}-\mathbf{P}\right) \mathbf{Z}$ is nonsingular.

Prove that if $\mathbf{X}_{1}$ is $n \times r$ of rank $r$ and consists of a set of $r$ linearly independent columns of $\mathbf{X}$, then $\mathbf{X} = \mathbf{X}_{1} \mathbf{L}$, where $\mathbf{L}$ is $r \times p$ of rank $r$.

Prove that $\mathbf{B}$ has full column rank.

If $\mathbf{X}$ has full rank and $\widehat{\mathbf{\theta }}_{\omega } = \mathbf{X} \widehat{\mathbf{\beta }}_{H}$, show that

Show how to modify the theory above to take care of the case when the restrictions are $\mathbf{A} \mathbf{\beta } = \mathrm{c}(\mathrm{c} \neq 0)$.

Let $Y_{i} = \beta x_{i}+\varepsilon_{i}(i = 1,2)$, where $\varepsilon_{1} \sim N\left(0, \sigma^{2}\right), \varepsilon_{2} \sim N\left(0,2 \sigma^{2}\right)$, and $\varepsilon_{1}$ and $\varepsilon_{2}$ are statistically independent. If $x_{1} = +1$ and $x_{2} = -1$, obtain the weighted least squares estimate of $\beta$ and find the variance of your estimate.

Let $Y_{i}(i = 1,2, \ldots , n)$ be independent random variables with a common mean $\theta$ and variances $\sigma^{2} / w_{i}(i = 1,2, \ldots , n)$. Find the linear unbiased estimate of $\theta$ with minimum variance, and find this minimum variance.

Let $Y_{1}, Y_{2}, \ldots , Y_{n}$ be independent random variables, and let $Y_{i}$ have a $N\left(i \theta , i^{2} \sigma^{2}\right)$ distribution for $i = 1,2, \ldots , n$. Find the weighted least squares estimate of $\theta$ and prove that its variance is $\sigma^{2} / n$.

Let $Y_{1}, Y_{2}, \ldots , Y_{n}$ be random variables with common mean $\theta$ and with dispersion matrix $\sigma^{2} \mathbf{V}$, where $v_{i i} = 1(i = 1,2, \ldots , n)$ and $v_{i j} = \rho$ ($0 < \rho < 1 ; i, j = 1,2, \ldots , n ; i \neq j$). Find the generalized least squares estimate of $\theta$ and show that it is the same as the ordinary least squares estimate. Hint: $\mathbf{V}^{-1}$ takes the same form as $\mathbf{V}$.

Let $\mathbf{Y} \sim N_{n}\left(\mathbf{X} \beta , \sigma^{2} \mathbf{V}\right)$, where $\mathbf{X}$ is $n \times p$ of $\operatorname {rank} p$ and $\mathbf{V}$ is a known positive-definite $n \times n$ matrix. If $\beta^{*}$ is the generalized least squares estimate of $\mathbf{\beta }$, prove that

Suppose that $\mathbb {E}\left[\mathbf{Y}\right] = \mathbf{\theta }, \mathbf{A} \mathbf{\theta } = \mathbf{0}$, and $\operatorname {Var}\left[\mathbf{Y}\right] = \sigma^{2} \mathbf{V}$, where $\mathbf{A}$ is a $q \times n$ matrix of $\operatorname {rank} q$ and $\mathbf{V}$ is a known $n \times n$ positive-definite matrix. Let $\mathbf{\theta }^{*}$ be the generalized least squares estimate of $\mathbf{\theta }$; that is, $\mathbf{\theta }^{*}$ minimizes $(\mathbf{Y}-\mathbf{\theta })^{T} \mathbf{V}^{-1}(\mathbf{Y}-\mathbf{\theta })$ subject to $\mathbf{A} \mathbf{\theta } = \mathbf{0}$. Show that

where $\tilde{\mathbf{\gamma }}^{*}$ is the generalized least squares estimate of $\mathbf{\gamma }$ for the model $\mathbb {E}\left[\mathbf{Y}\right] = \mathbf{V A}^{T} \mathbf{\gamma }, \operatorname {Var}\left[\mathbf{Y}\right] = \sigma^{2} \mathbf{V}$.

If $\tilde{Y}_{i} = Y_{i}-\bar{Y}$ and $\tilde{\mathbf{Y}} = \left(\tilde{Y}_{1}, \ldots , \tilde{Y}_{n}\right)^{T}$, prove from (3.54) that RSS $= \tilde{\mathbf{Y}}^{T}\left(\mathbf{I}_{n}-\tilde{\mathbf{P}}\right) \tilde{\mathbf{Y}}$.

Suppose that we consider fitting a model in which the $Y$-data are centered and scaled as well as the $x$-data. This means that we use $Y_{i}^{*} = \left(Y_{i}-\bar{Y}\right) / s_{y}$ instead of $Y_{i}$, where $s_{y}^{2} = \sum_{i}\left(Y_{i}-\bar{Y}\right)^{2}$. Using (3.54), obtain an expression for RSS from this model.

Derive equations (3.60) and (3.61).

Using the noninformative prior for $\theta$, show that the conditional posterior density $f(\beta \mid \mathbf{y}, \sigma )$ is multivariate normal. Hence deduce that the posterior mean of $\mathbf{\beta }$ is $\widehat{\mathbf{\beta }}$.

Suppose that we use the noninformative prior for $\mathbf{\theta }$.

Let $\chi (z) = \operatorname {sign}(\left|z\right|-1 / c)$ for some constant $c$. Show that the solution of (3.77) is the MAD estimate

Show that if we put $w_{i j} = 1$ in (3.99), we get Jaeckel's estimator defined by (3.97) with Wilcoxon weights.

Show that if $s$ is the solution of (3.101) with $\rho$ given by (3.102), then the resulting $s$ is approximately the lower quartile of the $\binom {n}{2}$ differences $\left|e_{i}(\mathbf{b})-e_{j}(\mathbf{b})\right|$.