Projections and Projection Matrices

Let $\mathbf{Y}$ represent a matrix in a linear space $\mathcal{V}$, let $\mathcal{U}$ and $\mathcal{W}$ represent subspaces of $\mathcal{V}$, and take $\{ \mathbf{X}_{1}, \ldots , \mathbf{X}_{s}\}$ to be a set of matrices that spans $\mathcal{U}$ and $\{ \mathbf{Z}_{1}, \ldots , \mathbf{Z}_{t}\}$ to be a set that spans $\mathcal{W}$. Verify that $\mathbf{Y} \perp \mathcal{U}$ if and only if $\mathbf{Y} \cdot \mathbf{X}_{i} = 0$ for $i = 1, \ldots , s$ (i.e., that $\mathbf{Y}$ is orthogonal to $\mathcal{U}$ if and only if $\mathbf{Y}$ is orthogonal to each of the matrices $\mathbf{X}_{1}, \ldots , \mathbf{X}_{s}$); and, similarly, that $\mathcal{U} \perp \mathcal{W}$ if and only if $\mathbf{X}_{i} \cdot \mathbf{Z}_{j} = 0$ for $i = 1, \ldots , s$ and $j = 1, \ldots , t$ (i.e., that $\mathcal{U}$ is orthogonal to $\mathcal{W}$ if and only if each of the matrices $\mathbf{X}_{1}, \ldots , \mathbf{X}_{s}$ is orthogonal to each of the matrices $\mathbf{Z}_{1}, \ldots , \mathbf{Z}_{t}$).

Let $\mathcal{U}$ and $\mathcal{V}$ represent subspaces of $\mathbb {R}^{m \times n}$. Show that if $\dim (\mathcal{V}) > \dim (\mathcal{U})$, then $\mathcal{V}$ contains a nonnull matrix that is orthogonal to $\mathcal{U}$.

Let $\mathcal{U}$ represent a subspace of the linear space $\mathbb {R}^{m}$ of all $m$-dimensional column vectors. Take $\mathcal{M}$ to be the subspace of $\mathbb {R}^{m \times n}$ defined by $\mathbf{W} \in \mathcal{M}$ if and only if $\mathbf{W} = (\mathbf{w}_{1}, \ldots , \mathbf{w}_{n})$ for some vectors $\mathbf{w}_{1}, \ldots , \mathbf{w}_{n}$ in $\mathcal{U}$. Let $\mathbf{Z}$ represent the projection (with respect to the usual inner product) of an $m \times n$ matrix $\mathbf{Y}$ on $\mathcal{M}$, and let $\mathbf{X}$ represent any $m \times p$ matrix whose columns span $\mathcal{U}$. Show that $\mathbf{Z} = \mathbf{X}\mathbf{B}^{*}$ for any solution $\mathbf{B}^{*}$ to the linear system

The projection (with respect to the usual inner product) of an $n$-dimensional column vector $\mathbf{y}$ on a subspace $\mathcal{U}$ of $\mathbb {R}^{n}$ in the special case where $n = 3$, $\mathbf{y} = (3, -38/5, 74/5)'$ and $\mathcal{U} = \operatorname {sp}\{ \mathbf{x}_{1}, \mathbf{x}_{2}, \mathbf{x}_{3}\}$, with

was determined to be the vector $(3, 22/5, 44/5)'$ --- and it was observed that $\mathbf{x}_{1}$ and $\mathbf{x}_{2}$ are linearly independent and that $\mathbf{x}_{3} = \mathbf{x}_{2} - (1/3)\mathbf{x}_{1}$, with the consequence that $\dim (\mathcal{U}) = 2$. Recompute the projection of $\mathbf{y}$ on $\mathcal{U}$ (in this special case) by taking $\mathbf{X}$ to be the $3 \times 2$ matrix

and carrying out the following two steps: (1) compute the solution to the normal equations $\mathbf{X}'\mathbf{X}\mathbf{b} = \mathbf{X}'\mathbf{y}$; and (2) postmultiply $\mathbf{X}$ by the solution you computed in Step (1).

Let $\mathbf{X}$ represent any $n \times p$ matrix. If a $p \times n$ matrix $\mathbf{B}^{*}$ is a solution to the linear system $\mathbf{X}'\mathbf{X}\mathbf{B} = \mathbf{X}'$ (in $\mathbf{B}$), then $\mathbf{B}^{*}$ is a generalized inverse of $\mathbf{X}$ and $\mathbf{X}\mathbf{B}^{*}$ is symmetric. Show that, conversely, if a $p \times n$ matrix $\mathbf{G}$ is a generalized inverse of $\mathbf{X}$ and if $\mathbf{X}\mathbf{G}$ is symmetric, then $\mathbf{X}'\mathbf{X}\mathbf{G} = \mathbf{X}'$ (i.e., $\mathbf{G}$ is a solution to $\mathbf{X}'\mathbf{X}\mathbf{B} = \mathbf{X}'$).

Using the result of Part (b) of Exercise 9.3 (or otherwise), show that, for any nonnull symmetric matrix $\mathbf{A}$,

where $\mathbf{B}$ is any matrix of full column rank and $\mathbf{T}$ any matrix of full row rank such that $\mathbf{A} = \mathbf{B}\mathbf{T}$. (That $\mathbf{T}\mathbf{B}$ is nonsingular follows from the result of Exercise 8.3.)

Let $\mathcal{V}$ represent a $k$-dimensional subspace of the linear space $\mathbb {R}^{n}$ of all $n$-dimensional column vectors. Take $\mathbf{X}$ to be any $n \times p$ matrix whose columns span $\mathcal{V}$, let $\mathcal{U}$ represent a subspace of $\mathcal{V}$, and define $\mathbf{A}$ to be the projection matrix for $\mathcal{U}$. Show (1) that a matrix $\mathbf{B}$ (of dimensions $n \times n$) is such that $\mathbf{B}\mathbf{y}$ is the projection of $\mathbf{y}$ on $\mathcal{U}$ for every $\mathbf{y} \in \mathcal{V}$ if and only if $\mathbf{B} = \mathbf{A} + \mathbf{Z}_{*}$ for some solution $\mathbf{Z}_{*}$ to the homogeneous linear system $\mathbf{X}'\mathbf{Z} = \mathbf{0}$ (in an $n \times n$ matrix $\mathbf{Z}$) and (2) that, unless $k = n$, there is more than one matrix $\mathbf{B}$ such that $\mathbf{B}\mathbf{y}$ is the projection of $\mathbf{y}$ on $\mathcal{U}$ for every $\mathbf{y} \in \mathcal{V}$.

Let $\{ \mathbf{A}_{1}, \ldots , \mathbf{A}_{k}\}$ represent a nonempty linearly independent set of matrices in a linear space $\mathcal{V}$. And, define (as in Gram-Schmidt orthogonalization) $k$ nonnull orthogonal linear combinations, say $\mathbf{B}_{1}, \ldots , \mathbf{B}_{k}$, of $\mathbf{A}_{1}, \ldots , \mathbf{A}_{k}$ as follows:

where (for $i < j = 1, \ldots , k$)

Show that $\mathbf{B}_{j}$ is the (orthogonal) projection of $\mathbf{A}_{j}$ on some subspace $\mathcal{U}_{j}$ (of $\mathcal{V}$) and describe $\mathcal{U}_{j}$ ($j = 2, \ldots , k$).