Norms, Inner Products, and Orthogonality

Find the 1-, 2-, and $\infty$-norms of $x = \begin{pmatrix} 2 \\ 1 \\ -4 \\ -2 \end{pmatrix}$ and $x = \begin{pmatrix} 1+i \\ 1-i \\ 1 \\ 4i \end{pmatrix}$.

Consider the euclidean norm with $u = \begin{pmatrix} 2 \\ 1 \\ -4 \\ -2 \end{pmatrix}$ and $v = \begin{pmatrix} 1 \\ -1 \\ 1 \\ -1 \end{pmatrix}$.

Show that $\left(\alpha_{1} + \alpha_{2} + \cdots + \alpha_{n}\right)^{2} \leq n \left(\alpha_{1}^{2} + \alpha_{2}^{2} + \cdots + \alpha_{n}^{2}\right)$ for $\alpha_{i} \in \mathbb {R}$.

If $x, y \in \mathbb {R}^{n}$ such that $\left\| x-y\right\|_{2} = \left\| x+y\right\|_{2}$, what is $x^{T}y$?

Explain why $\left\| x-y\right\| = \left\| y-x\right\|$ is true for all norms.

For every vector norm on $\mathbb {C}^{n}$, prove that $\left\| v\right\|$ depends continuously on the components of $v$ in the sense that for each $\varepsilon > 0$, there corresponds a $\delta > 0$ such that $\left|\left\| x\right\| - \left\| y\right\| \right| < \varepsilon$ whenever $\left|x_{i} - y_{i}\right| < \delta$ for each $i$.

For $x, y \in \mathbb {C}^{n}$, $x \neq 0$, explain why equality holds in the CBS inequality if and only if $y = \alpha x$, where $\alpha = x^{*}y / x^{*}x$.

For nonzero vectors $x, y \in \mathbb {C}^{n}$ with the euclidean norm, prove that equality holds in the triangle inequality if and only if $y = \alpha x$, where $\alpha$ is real and positive.

Use Holder's inequality

(where $p>1$, $q>1$, $1/p+1/q=1$) to prove that if the components of $x \in \mathbb {R}^{n \times 1}$ sum to zero (i.e., $x^{T}e = 0$ for $e^{T} = (1,1,\ldots ,1)$), then

for all $y \in \mathbb {R}^{n \times 1}$.

The classical form of Holder's inequality states that if $p>1$ and $q>1$ are real numbers such that $1/p+1/q=1$, then

Derive this inequality by executing the following steps:

The triangle inequality $\left\| x+y\right\|_{p} \leq \left\| x\right\|_{p}+\left\| y\right\|_{p}$ for a general $p$-norm is really the classical Minkowski inequality, which states that for $p \geq 1$,

Derive Minkowski's inequality.

Evaluate the Frobenius matrix norm for each matrix below.

Evaluate the induced 1-, 2-, and $\infty$-matrix norm for each of the three matrices given in Exercise 5.2.1: $A = \begin{pmatrix} 1 & -2 \\ -1 & 2 \end{pmatrix}$, $B = \begin{pmatrix} 0 & 1 & 0 \\ 0 & 0 & 1 \\ 1 & 0 & 0 \end{pmatrix}$, $C = \begin{pmatrix} 4 & -2 & 4 \\ -2 & 1 & -2 \\ 4 & -2 & 4 \end{pmatrix}$.

Explain why $\left\| A\right\|_{F} = \left\| A^{*}\right\|_{F}$ for the Frobenius matrix norm $\left\| A\right\|_{F}^{2} = \sum_{i,j} \left|a_{ij}\right|^{2} = \operatorname {trace}(A^{*}A)$.

For matrices $A$ and $B$ and for vectors $x$, establish the following compatibility properties between a vector norm defined on every $\mathbb {C}^{p}$ and the associated induced matrix norm $\left\| A\right\| = \max_{\left\| x\right\| =1} \left\| Ax\right\|$.

Establish the following properties of the matrix 2-norm.

Using the induced matrix norm $\left\| A\right\| = \max_{\left\| x\right\| =1} \left\| Ax\right\|$, prove that if $A$ is nonsingular, then

For $A \in \mathbb {C}^{n \times n}$ and a parameter $z \in \mathbb {C}$, the matrix $R(z) = (zI-A)^{-1}$ is called the resolvent of $A$. Prove that if $\left|z\right| > \left\| A\right\|$ for any induced matrix norm, then

For $x = \begin{pmatrix} x_{1} \\ x_{2} \\ x_{3} \end{pmatrix}$, $y = \begin{pmatrix} y_{1} \\ y_{2} \\ y_{3} \end{pmatrix}$, determine which of the following are inner products for $\mathbb {R}^{3 \times 1}$.

For a general inner-product space $V$, explain why each of the following statements must be true.

Let $V$ be an inner-product space with an inner product $\left\langle x, y \right\rangle$. Explain why the function defined by $\left\| \star \right\| = \sqrt{ \left\langle \star , \star \right\rangle }$ satisfies the first two general-matrix-norm properties $\left\| A\right\| \geq 0$ with $\left\| A\right\| =0 \iff A=0$, and $\left\| \alpha A\right\| = \left|\alpha \right|\left\| A\right\|$.

For a real inner-product space with $\left\| \star \right\|^{2} = \left\langle \star , \star \right\rangle$, derive the inequality

For $n \times n$ matrices $A$ and $B$, explain why each of the following inequalities is valid.

Extend the proof of the parallelogram identity's converse to include complex spaces.

Explain why there does not exist an inner product on $\mathbb {C}^{n}$ ($n \geq 2$) such that $\left\| \star \right\|_{\infty } = \sqrt{ \left\langle \star , \star \right\rangle }$.

Explain why the Frobenius matrix norm on $\mathbb {C}^{n \times n}$ must satisfy the parallelogram identity.

For $n \geq 2$, is either the matrix 1-, 2-, or $\infty$-norm generated by an inner product on $\mathbb {C}^{n \times n}$?

Using the standard inner product, determine which of the following pairs are orthogonal vectors in the indicated space.

Find two vectors of unit norm that are orthogonal to $u = \begin{pmatrix} 3 \\ -2 \end{pmatrix}$.

Consider the following set of three vectors:

Using the standard inner product, determine the Fourier expansion of $x$ with respect to $B$, where

With respect to the inner product for matrices $\left\langle A, B \right\rangle = \operatorname {trace}(A^{T}B)$, verify that the set

is an orthonormal basis for $\mathbb {R}^{2 \times 2}$, and then compute the Fourier expansion of $A = \begin{pmatrix} 1 & 1 \\ 1 & 1 \end{pmatrix}$ with respect to $B$.

Determine the angle between $x = \begin{pmatrix} 2 \\ -1 \\ 1 \end{pmatrix}$ and $y = \begin{pmatrix} 1 \\ 1 \\ 2 \end{pmatrix}$.

Given an orthonormal basis $B$ for a space $V$, explain why the Fourier expansion for $x \in V$ is uniquely determined by $B$.

Explain why the columns of $U_{n \times n}$ are an orthonormal basis for $\mathbb {C}^{n}$ if and only if $U^{*} = U^{-1}$.

Matrices with the property $A^{*}A = AA^{*}$ are said to be normal (hermitian matrices as well as real symmetric matrices are included in this class). Prove that if $A$ is normal, then $\mathcal{R}(A) \perp \mathcal{N}(A)$ — i.e., every vector in the range of $A$ is orthogonal to every vector in the nullspace of $A$.

Using the trace inner product $\left\langle A, B \right\rangle = \operatorname {trace}(A^{T}B)$, determine the angle between the following pairs of matrices.

Why is the definition for $\cos \theta = \left\langle x, y \right\rangle /\left\| x\right\| \left\| y\right\|$ not good for $\mathbb {C}^{n}$? Explain how to define $\cos \theta$ so that it makes sense in $\mathbb {C}^{n}$.

If $\left\{ u_{1}, u_{2}, \ldots , u_{n}\right\}$ is an orthonormal basis for an inner-product space $V$, explain why

holds for every $x, y \in V$.

Consider a real inner-product space, where $\left\| \star \right\|^{2} = \left\langle \star , \star \right\rangle$.

Let $V$ be a general inner-product space in which $\left\| \star \right\|^{2} = \left\langle \star , \star \right\rangle$.

Let $B = \left\{ u_{1}, u_{2}, \ldots , u_{n}\right\}$ be an orthonormal basis for an inner-product space $V$, and let $x = \sum_{i} \xi_{i}u_{i}$ be the Fourier expansion of $x \in V$.

Let $B = \left\{ u_{1}, u_{2}, \ldots , u_{k}\right\}$ be an orthonormal set in an $n$-dimensional inner-product space $V$. Derive Bessel's inequality, which says that if $x \in V$ and $\xi_{i} = \left\langle u_{i}, x \right\rangle$, then

Explain why equality holds if and only if $x \in \left\langle u_{1}, u_{2}, \ldots , u_{k}\right\rangle$.

Construct an example using the standard inner product in $\mathbb {R}^{n}$ to show that two vectors $x$ and $y$ can have an angle between them that is close to $\pi /2$ without $x^{T}y$ being close to $0$.

Example 5.4.3 shows that $y$ is linearly correlated with $x$ in the sense that $y \approx \beta_{0}e + \beta_{1}x$ if and only if the standardization vectors $z_{x} = (x-\mu_{x}e)/\sigma_{x}$ and $z_{y} = (y-\mu_{y}e)/\sigma_{y}$ are "close" in the sense that they are almost on the same line in $\mathbb {R}^{n}$ (where $\mu_{x}, \sigma_{x}$ are the mean and standard deviation of $x$'s entries, and similarly for $y$). Explain why simply measuring $\left\| z_{x}-z_{y}\right\|_{2}$ does not always gauge the degree of linear correlation.

Let $\theta$ be the angle between two vectors $x$ and $y$ from a real inner-product space.

With respect to the orthonormal set

determine the Fourier series expansion of the saw-toothed function defined by $f(t) = t$ for $-\pi < t < \pi$ (extended periodically).

The periodic extension of the saw-toothed function f(t)=t on (-\pi ,\pi ).

Let $S = \left\langle x_{1} = \begin{pmatrix} 1 \\ 1 \\ 1 \\ -1 \end{pmatrix}, \; x_{2} = \begin{pmatrix} 2 \\ -1 \\ -1 \\ 1 \end{pmatrix}, \; x_{3} = \begin{pmatrix} -1 \\ 2 \\ 2 \\ 1 \end{pmatrix}\right\rangle$.

Use the Gram-Schmidt procedure to find an orthonormal basis for the four fundamental subspaces of $A = \begin{pmatrix} 1 & -2 & 3 & -1 \\ 2 & -4 & 6 & -2 \\ 3 & -6 & 9 & -3 \end{pmatrix}$.

Apply the Gram-Schmidt procedure with the standard inner product for $\mathbb {C}^{3}$ to $\left\{ \begin{pmatrix} i \\ i \\ i \end{pmatrix}, \begin{pmatrix} 0 \\ i \\ i \end{pmatrix}, \begin{pmatrix} 0 \\ 0 \\ i \end{pmatrix}\right\}$.

Explain what happens when the Gram-Schmidt process is applied to an orthonormal set of vectors.

Explain what happens when the Gram-Schmidt process is applied to a linearly dependent set of vectors.

Let $A = \begin{pmatrix} 1 & 0 & -1 \\ 1 & 2 & 1 \\ 1 & 1 & -3 \\ 0 & 1 & 1 \end{pmatrix}$ and $b = \begin{pmatrix} 1 \\ 1 \\ 1 \\ 1 \end{pmatrix}$.

Given a linearly independent set of vectors $S = \left\{ x_{1},x_{2},\ldots ,x_{n}\right\}$ in an inner-product space, let $S_{k} = \left\langle x_{1},x_{2},\ldots ,x_{k}\right\rangle$ for $k=1,2,\ldots ,n$. Give an induction argument to prove that if $O_{k} = \left\{ u_{1},u_{2},\ldots ,u_{k}\right\}$ is the Gram-Schmidt sequence

then $O_{k}$ is indeed an orthonormal basis for $S_{k} = \left\langle x_{1},x_{2},\ldots ,x_{k}\right\rangle$ for each $k=1,2,\ldots ,n$.

Prove that if $\operatorname {rk}\left(A_{m \times n}\right) = n$, then the rectangular QR factorization of $A$ is unique. That is, if $A = QR$, where $Q_{m \times n}$ has orthonormal columns and $R_{n \times n}$ is upper triangular with positive diagonal entries, then $Q$ and $R$ are unique.