Prediction and Model Selection

Explain how you could calculate the ridge estimate $\widehat{\beta }(k)$ using a least squares program.

Consider a regression with two orthogonal centered and scaled explanatory variables $x$ and $z$. Using the techniques of Example 12.2 in Section 12.4.1, calculate the probability of correctly identifying the true model using forward selection when the regression coefficients of $x$ and $z$ are both zero. Use a value of $c_{1} = 1.64$ and assume that the error variance is 1.

Consider a regression with $K$ explanatory variables. Let $F_{p}$ be the $F$-statistic for testing that some specified subset of $r$ variables can be deleted from the model, where $r = K+1-p$. Show that the value of $C_{p}$ for this reduced model with the $r$ variables deleted is $C_{p} = r\left(F_{p}-1\right)+p$.

Verify (12.85).

Show that the square of the total bias $\left\| \mathbf{\mu }-\mathbb {E}\left[\mathbf{X} \widehat{\mathbf{\beta }}\right]\right\|^{2}$ is equal to the first term in (12.8).

Consider the regression model

where $\sum_{i} x_{i} = \sum_{i} z_{i} = 0$ and $\sum_{i} x_{i}^{2} = \sum_{i} z_{i}^{2} = 1$. Suppose that we want to predict the response $Y$ corresponding to a vector $(x, z)$ using a predictor of the form

where $\tilde{\gamma }_{1}$ is the least squares estimate obtained by fitting the model

Prove that

Perform a small simulation to assess how badly $C_{p}$ underestimates the prediction error in Example 12.1.

Prove that $K L(f, g) \geq 0$. You may assume that $f(\mathbf{y}) > 0$ and $g(\mathbf{y}) > 0$ for all x . Hint: Show that $\log x \leq x-1$ for $x > 0$, and put $x = g(\mathbf{y}) / f(\mathbf{y})$.

Obtain (12.35) using conditional probability arguments.

Show that the first step in forward selection is equivalent to selecting the variable most highly correlated with the response.

Show that the variable that increases the difference $\mathrm{RSS}_{p}-\mathrm{RSS}_{p+1}$ by the greatest amount is the one having the greatest partial correlation with the response, given the variables already in the model.

In Example 12.2, verify Algorithm 12.2 and identify the regions corresponding to the subsets chosen by (a) backward elimination, (b) stepwise regression.

Verify that the marginal distribution of $\mathbf{Y}$ in the Bayesian setup of Section 12.5.1 is $N_{p}\left(\mathbf{0},\left(\sigma^{2}+\sigma_{0}^{2}\right) \mathbf{I}_{p}\right)$.

Show that when $\mathbf{X}^{T} \mathbf{X} = \mathbf{I}_{p}$, then the quantity $c_{j}$ in the second version of the garrote is given by

Fig. 12.4 Functions giving shrinkage estimates in the orthonormal case. The shrinkage estimate \tilde{\beta }{j} is h\left(\widehat{\beta }{j}\right).

Verify (12.65). Hint: Use the partitioned matrix inverse formula A.9.1 and multiply out the quadratic form.

Find the form of the posterior predictive density for straight-line regression. Assume that the explanatory variable has been centered and that the prior mean is zero. Also assume that you are predicting a single value (i.e., assume that $m = 1$).

Consider the regression model

where $\sum_{i} x_{i} = 0$ and $\sum_{i} x_{i}^{2} = 1$. Let $r = \sum_{i} x_{i} Y_{i}$. Suppose that we estimate the parameter $\beta$ by the estimate

where $c$ is some positive constant. How does the MSE of $\tilde{\beta }$ as an estimate of $\beta$ compare with that of the usual LSE?

Suppose that you use forward selection to choose either the null model $Y_{i} = \alpha +\varepsilon_{i}$ or the simple linear regression model in Exercise 1 above. Explain how you would evaluate the conditional density (12.74) in this situation.

For any pair of random vectors $\mathbf{Y}$ and $\mathbf{Z}$, show that the difference $\operatorname {Var}\left[\mathbf{Y}\right]-\mathbb {E}\left[\operatorname {Var}\left[\mathbf{Y} \mid \mathbf{Z}\right]\right]$ is positive semidefinite. Hint: Show that the difference is $\operatorname {Var}\left[\mathbb {E}\left[\mathbf{Y} \mid \mathbf{Z}\right]\right]$.

Write a small computer program to calculate the residual sum of squares for all possible regressions using the Garside algorithm. What size problems can reasonably be handled?

Implement the Morgan and Tatar algorithm. What sort of improvements do you notice over the Garside algorithm?

Show that the function $f$ which minimizes the expected ME (12.80) is the conditional expectation $\mathbb {E}\left[\beta \mid \widehat{\beta }\right]$ given by (12.81).

Draw the same diagram for Example 12.6 as those given in the text in Examples 12.4 and 12.5.

Show that in ridge regression when $\mathbf{X}^{T} \mathbf{X} = \mathbf{I}_{p}$, the little bootstrap and GCV give the same estimate for the ridge parameter.