Departures from Assumptions: Diagnosis and Remedies

Show that

Let $\eta_{i j} = \left[\left(\mathbf{I}_{n}-\mathbf{H}_{j}\right) \mathbf{x}^{(j)}\right]_{i}$. Show that if $h_{i}$ and $h_{i}^{(j)}$ are the diagonal elements of $\mathbf{H}$ and $\mathbf{H}_{j}$, then

Explain how the second term on the right of (10.72) can be interpreted as the leverage of the $i$ th case in the added variable plot for variable $j$. How can you use added variable plots to determine which variables contribute to the leverage of observation $j$?

Assume that $\mathbf{X}$ is of full rank $p$. Let $\mathbf{Q} = \left(\mathbf{Q}_{1}, \mathbf{Q}_{2}\right)$ be an $n \times n$ orthogonal matrix such that the columns of $\mathbf{Q}_{1}$ are an orthonormal basis for $\mathcal{C}(\mathbf{X})$. (For example, $\mathbf{Q}$ could be the matrix arising in the QR decomposition discussed in Section 11.3.) Show that any matrix $\mathbf{A}$ that satisfies (ii) and (iii) above must be of the form $\mathbf{T Q}_{2}^{T}$ for some $(n-p) \times (n-p)$ orthogonal matrix $\mathbf{T}$.

Let $\mathbf{J}$ be an $(n-p) \times n$ submatrix of $\mathbf{I}_{n}$ such that $\varepsilon_{1} = \mathbf{J} \varepsilon$. Show that

Let $\mathbf{Q}_{2}^{T} \mathbf{J}^{T} = \mathbf{U} \mathbf{\Delta } \mathbf{V}^{T}$ be the singular value decomposition of $\mathbf{Q}_{2}^{T} \mathbf{J}^{T}$. (cf. A.12). Show that

with equality if and only if $\mathbf{T} = \mathbf{V} \mathbf{U}^{T}$. Hence show that $\mathbf{A} = \mathbf{V} \mathbf{U}^{T} \mathbf{Q}_{2}^{T}$. Hint: Show that $\operatorname {tr}\left[(\mathbf{V}-\mathbf{T U}) \mathbf{\Delta }(\mathbf{V}-\mathbf{T U})^{T}\right] = 2\left[\operatorname {tr}(\mathbf{\Delta })-\operatorname {tr}\left(\mathbf{T} \mathbf{Q}_{2}^{T} \mathbf{J}^{T}\right)\right]$.

Express $e_{i}$ in the form $e_{i} = \mathbf{c}_{i}^{T}\left(\mathbf{I}_{n}-\mathbf{H}\right) \varepsilon$ for a suitable vector $\mathbf{c}_{i}$.

Show that $(n-p)^{-1} r_{i}^{2}$ can be written as

where $\mathbf{Q} = \left(1-h_{i}\right)^{-1}\left(\mathbf{I}_{n}-\mathbf{H}\right) \mathbf{c}_{i} \mathbf{c}_{i}^{T}\left(\mathbf{I}_{n}-\mathbf{H}\right)$ and $\mathbf{Z} \sim N_{n}\left(\mathbf{0}, \mathbf{I}_{n}\right)$.

Prove that $\mathbf{Q}$ is a projection matrix (i.e., show that $\mathbf{Q}^{2} = \mathbf{Q}$ and $\mathbf{Q}^{T} = \mathbf{Q}$).

Show that $\left(\mathbf{I}_{n}-\mathbf{H}\right)-\mathbf{Q}$ is a projection matrix and prove that $\mathbf{Z}^{T} \mathbf{Q Z}$ and $\mathbf{Z}^{T}\left(\left(\mathbf{I}_{n}-\mathbf{H}\right)-\mathbf{Q}\right) \mathbf{Z}$ are independent. Hence prove that $(n-p)^{-1} r_{i}^{2}$ has a beta $\left[\frac{1}{2}, \frac{1}{2}(n-p-1)\right]$ distribution.

Show that provided that the regression matrix $\mathbf{X}$ contains a column of 1 's, $\sum_{i = 1}^{n}\left(x_{i j}-\bar{x}_{j}\right) e_{i} = 0$ for each $j$.

Use part (a) to prove (10.21).

The following exercises require familiarity with a computing package such as S-PLUS, R, or Matlab. We provide some R code to assist readers.

Generate a set of regression data according to the model

where $x_{i 1}$ and $x_{i 2}$ are sampled from independent $N(0,1)$ distributions, and the errors $\varepsilon_{i}$ are sampled from $N\left(0, \sigma^{2}\right)$, where $\sigma = 0.25$. This may be done in S-PLUS or R by the code fragment eps<-rnorm(100,sd=0.25) x1<-rnorm(100) x2<-rnorm(100) y<-1+x1+2*sign(x2)*abs(x2)^(1/3)+eps

Construct the partial residual plot for $x_{2}$. How well does the curve in the plot reveal the transformation required to linearize the regression surface? Use the code reg.stuff<-lm(y~x1+x2) betahat2<-coef(reg.stuff)[3] estar2<-betahat2*x2+residuals(reg.stuff) plot(x2,estar2)

Now construct another set of data, the same as before but with $x_{2}$ having correlation $\rho = 0.95$ with $x_{1}$. Use the same code, but define $x_{2}$ with the lines rho<-0.95 x2<-rho*x1+sqrt(1-rho^2)*rnorm(100)

Construct the partial residual plot for the new $x_{2}$. What do you notice?

Let $\mathbf{G} = \mathbf{\Sigma }^{-1} \mathbf{X}\left(\mathbf{X}^{T} \mathbf{\Sigma }^{-1} \mathbf{X}\right)^{-1}$, and let $\mathbf{Q}$ be an orthonormal basis for $\mathcal{C}(\mathbf{X})^{\perp }$, so that $\mathbf{Q} \mathbf{Q}^{T} = \mathbf{I}_{n}-\mathbf{X}\left(\mathbf{X}^{T} \mathbf{X}\right)^{-1} \mathbf{X}^{T}$. Verify that $\mathbf{G}^{T} \mathbf{\Sigma } \mathbf{Q} = \mathbf{0}$ and that $\mathbf{G}^{T} \mathbf{\Sigma } \mathbf{G} = \left(\mathbf{X}^{T} \mathbf{\Sigma }^{-1} \mathbf{X}\right)^{-1}$.

Let $\mathbf{M} = \left(\mathbf{\Sigma }^{1 / 2} \mathbf{Q}, \mathbf{\Sigma }^{1 / 2} \mathbf{G}\right)$. Show that $\mathbf{M}$ is of full rank, so that $\mathbf{M}\left(\mathbf{M}^{T} \mathbf{M}\right)^{-1} \mathbf{M}^{T} = \mathbf{I}_{n}$.

Use (b) to prove (10.38).

Use the fact that $[\operatorname {det}\left(\mathbf{M}\right)]^{2} = \operatorname {det}\left(\mathbf{M}^{T} \mathbf{M}\right)$ to prove (10.39).

Suppose that $\mathbf{Y} = \left(Y_{1}, \ldots , Y_{n}\right)^{T}$, where the $Y_{i}$ 's are independently and identically distributed as $N\left(\mu , \sigma^{2}\right)$. Find the restricted likelihood for $\mathbf{Q}^{T} \mathbf{Y}$.

Show that for this example the REML estimate of $\sigma^{2}$ is the usual unbiased estimate $S^{2} = (n-1)^{-1} \sum_{i = 1}^{n}\left(Y_{i}-\bar{Y}\right)^{2}$. Thus in this case the REML estimate is unbiased but the MLE is not.

By making a transformation, show that

where $B(i, n-i+1)$ is the beta function (A.13.6).

Using the definition of the integral, show that

where

Explain why the weights $w_{i j}$ sum to 1.

Comment on the accuracy of the approximation

Show that the log likelihood for $Y_{1}, \ldots , Y_{n}$ is

Show that in the case of the John-Draper transformation, the log likelihood is

Show that $(n-1)^{-1}\left(n \widehat{F}_{n}-\delta_{i}\right)$ is the empirical distribution function calculated from the remaining $(n-1)$ points when the $i$ th observation has been deleted from the sample.

Hence show that for the least squares estimate functional $T$,

The quantity $\mathrm{SIC}_{i}$ is a vector. We can obtain an overall scalar measure by considering the quantity

where $\mathbf{M}$ is a positive-definite matrix and $c$ is a positive constant. Show that if we choose $\mathbf{M} = \mathbf{X}^{T} \mathbf{X}$ and $c = p S^{2} /(n-1)^{2}$, we obtain Cook's D.

For more details on the influence function approach to outlier detection, see Cook and Weisberg [1982, Chapter 3] and Chatterjee and Hadi [1988, Chapter 5].