3.1 Least Squares Estimation

Show that if $\mathbf{X}$ has full rank,

and hence deduce that the left side is minimized uniquely when $\mathbf{\beta } = \widehat{\mathbf{\beta }}$.

If $\mathbf{X}$ has full rank, prove that $\sum_{i = 1}^{n}\left(Y_{i}-\widehat{Y}_{i}\right) = 0$. Hint: Consider the first column of $\mathbf{X}$.

Let

where $\mathbb {E}\left[\varepsilon_{i}\right] = 0(i = 1,2,3)$. Find the least squares estimates of $\theta$ and $\phi$.

Consider the regression model

where $x_{1} = -1, x_{2} = 0$, and $x_{3} = +1$. Find the least squares estimates of $\beta_{0}, \beta_{1}$, and $\beta_{2}$. Show that the least squares estimates of $\beta_{0}$ and $\beta_{1}$ are unchanged if $\beta_{2} = 0$.

The tension $T$ observed in a nonextensible string required to maintain a body of unknown weight $w$ in equilibrium on a smooth inclined plane of angle $\theta (0 < \theta < \pi / 2)$ is a random variable with mean $\mathbb {E}\left[T\right] = w \sin \theta$. If for $\theta = \theta_{i}(i = 1,2, \ldots , n)$ the corresponding values of $T$ are $T_{i}$ ($i = 1,2, \ldots , n$), find the least squares estimate of $w$.

If $\mathbf{X}$ has full rank, so that $\mathbf{P} = \mathbf{X}\left(\mathbf{X}^{T} \mathbf{X}\right)^{-1} \mathbf{X}^{T}$, prove that $\mathcal{C}(\mathbf{P}) = \mathcal{C}(\mathbf{X})$.

For a general regression model in which $\mathbf{X}$ may or may not have full rank, show that

Suppose that we scale the explanatory variables so that $x_{i j} = k_{j} w_{i j}$ for all $i, j$. By expressing $\mathbf{X}$ in terms of a new matrix $\mathbf{W}$, prove that $\widehat{\mathbf{Y}}$ remains unchanged under this change of scale.