Kaczmarz's and Cimmino's projection methods

[2/100%]
Show
LaTeX

For a consistent system An×rx=bA_{n\times r}x=b with rk⁡(A)=r\operatorname {rk}\left(A\right)=r, scale the rows so that ∥Ai∗∥2=1\left\| A_{i*}\right\|_{2}=1 for each ii, and let Hi={x∣Ai∗x=bi}H_{i} = \left\{ x \mid A_{i*}x=b_{i}\right\} be the hyperplane defined by the iith equation. Beginning with an arbitrary vector p0p_{0}, the Kaczmarz sequence is generated by the double loop For k=0,1,2,3,…k=0,1,2,3,\ldots For i=1,2,…,ni=1,2,\ldots ,n pkn+i=pkn+i−1−(Ai∗pkn+i−1−bi)(Ai∗)Tp_{kn+i} = p_{kn+i-1} - (A_{i*}p_{kn+i-1}-b_{i})(A_{i*})^{T} Prove that the Kaczmarz sequence converges to the solution xx of Ax=bAx=b by showing

∥pkn+i−x∥22=∥pkn+i−1−x∥22−(Ai∗pkn+i−1−bi)2. \left\| p_{kn+i}-x\right\| _{2}^{2} = \left\| p_{kn+i-1}-x\right\| _{2}^{2} - (A_{i*}p_{kn+i-1}-b_{i})^{2}.
?

In 1938 the Italian mathematician Gianfranco Cimmino used the following elementary observation to construct an iterative algorithm for solving linear systems. For a 2×22\times 2 system Ax=bAx=b, let H1H_{1} and H2H_{2} be the two lines (hyperplanes) defined by the two equations. For an arbitrary guess r0r_{0}, let r1r_{1} be the reflection of r0r_{0} about the line H1H_{1}, and let r2r_{2} be the reflection of r0r_{0} about the line H2H_{2}. As illustrated below, the three points r0r_{0}, r1r_{1}, and r2r_{2} lie on a circle whose center is H1∩H2H_{1}\cap H_{2} (the solution of the system).

The reflections r_1, r_2 of an arbitrary point r_0 about lines \mathcal{H}_1, \mathcal{H}_2 lie, together with r_0, on a circle centered at \mathcal{H}_1 \cap \mathcal{H}_2.The reflections r_1, r_2 of an arbitrary point r_0 about lines \mathcal{H}_1, \mathcal{H}_2 lie, together with r_0, on a circle centered at \mathcal{H}_1 \cap \mathcal{H}_2.

The mean value m=(r1+r2)/2m = (r_{1}+r_{2})/2 is strictly inside the circle, so mm is a better approximation to the solution than r0r_{0}. It's visually evident that iteration produces a sequence that converges to the solution of Ax=bAx=b. Prove this in general by using the following blueprint.

?
(a)

For a scalar β\beta and a vector u∈Rnu \in \mathbb {R}^{n} such that ∥u∥2=1\left\| u\right\|_{2}=1, consider the hyperplane H={x∣uTx=β}H = \left\{ x \mid u^{T}x=\beta \right\} (Exercise 5.13.17). Show that the reflection of a vector bb about HH is r=b−2(uTb−β)ur = b - 2(u^{T}b-\beta )u.

(b)

For a system Ax=bAx=b in which the rows of A∈Rn×rA \in \mathbb {R}^{n\times r} have been scaled so that ∥Ai∗∥2=1\left\| A_{i*}\right\|_{2}=1 for each ii, let Hi={x∣Ai∗x=bi}H_{i} = \left\{ x \mid A_{i*}x=b_{i}\right\}. If r0∈Rrr_{0} \in \mathbb {R}^{r} is arbitrary, and rir_{i} is the reflection of r0r_{0} about HiH_{i}, explain why the mean value of the reflections {r1,…,rn}\left\{ r_{1},\ldots ,r_{n}\right\} is m=r0−(2/n)ATεm = r_{0} - (2/n)A^{T}\varepsilon, where ε=Ar0−b\varepsilon = Ar_{0}-b.

(c)

Iterating part (b) produces mk=mk−1−(2/n)ATεk−1m_{k} = m_{k-1} - (2/n)A^{T}\varepsilon_{k-1}, where εk−1=Amk−1−b\varepsilon_{k-1} = Am_{k-1}-b. Show that if AA is nonsingular, and x=A−1bx=A^{-1}b, then x−mk=(I−(2/n)ATA)k(x−m0)x-m_{k} = \left(I-(2/n)A^{T}A\right)^{k}(x-m_{0}).

Remark.
?

It can be proven that (I−(2/n)ATA)k→0\left(I-(2/n)A^{T}A\right)^{k} \to 0 as k→∞k\to \infty, so mk→xm_{k}\to x for all m0m_{0}, even if AA is rank deficient (converging to a solution if consistent, else a least squares solution). Cimmino's method also works with weighted means: if W=diag⁡(w1,…,wn)W = \operatorname {diag}(w_{1},\ldots ,w_{n}), wi>0w_{i}>0, ∑wi=1\sum w_{i}=1, then mk=mk−1−ωATWεk−1m_{k} = m_{k-1} - \omega A^{T}W\varepsilon_{k-1} converges for a relaxation parameter 0<ω<20<\omega <2.

References

[Mey00] Meyer C.D. Matrix Analysis and Applied Linear Algebra. SIAM, 2000.