Transcript MTH285-CH6.5-lecture

6 Orthogonality and Least
Squares
6.5
LEAST-SQUARES PROBLEMS
© 2012 Pearson Education, Inc.
LEAST-SQUARES PROBLEMS
 Definition: If A is m  n and b is in
, a leastn
ˆ
squares solution of Ax  b is an x in
such that
m
b  Axˆ  b  Ax
for all x in
n
.
 The most important aspect of the least-squares
problem is that no matter what x we select, the vector
Ax will necessarily be in the column space, Col A.
 So we seek an x that makes Ax the closest point in
Col A to b. See the figure on the next slide.
© 2012 Pearson Education, Inc.
Slide 6.5- 2
LEAST-SQUARES PROBLEMS
 Solution of the General Least-Squares Problem
 Given A and b, apply the Best Approximation
Theorem to the subspace Col A.
 Let
© 2012 Pearson Education, Inc.
bˆ  projCol Ab
Slide 6.5- 3
SOLUTION OF THE GENREAL LEAST-SQUARES
PROBLEM
ˆ
 Because bˆ is in the column space A, the equation Ax  b
n
ˆ
is consistent, and there is an x in such that
Axˆ  bˆ
----(1)
 Since bˆ is the closest point in Col A to b, a vector xˆ is a
least-squares solution of Ax  b if and only if xˆ
satisfies (1).
is a list of weights that will build bˆ out
of the columns of A. See the figure on the next slide.
 Such an xˆ in
© 2012 Pearson Education, Inc.
n
Slide 6.5- 4
SOLUTION OF THE GENREAL LEAST-SQUARES
PROBLEM
ˆ.
 Suppose xˆ satisfies Axˆ  b
 By the Orthogonal Decomposition Theorem, the
ˆ is orthogonal
projection bˆ has the property that b  b
ˆ is orthogonal to each column of A.
to Col A, so b  Ax
ˆ  0,
 If aj is any column of A, then a j (b  Ax)
T
ˆ .
and a j (b  Ax)
© 2012 Pearson Education, Inc.
Slide 6.5- 5
SOLUTION OF THE GENREAL LEAST-SQUARES
PROBLEM
 Since each a Tj is a row of AT,
ˆ 0
AT (b  Ax)
 Thus
----(2)
AT b  AT Axˆ  0
T
ˆ
A Ax  A b
T
 These calculations show that each least-squares
solution of Ax  b satisfies the equation
T
T
----(3)
A Ax  A b
 The matrix equation (3) represents a system of
equations called the normal equations for Ax  b.
 A solution of (3) is often denoted by xˆ .
© 2012 Pearson Education, Inc.
Slide 6.5- 6
SOLUTION OF THE GENREAL LEAST-SQUARES
PROBLEM
 Theorem 13: The set of least-squares solutions of Ax  b
coincides with the nonempty set of solutions of the
T
T
normal equation A Ax  A b.
 Proof: The set of least-squares solutions is nonempty and
each least-squares solution xˆ satisfies the normal
equations.
 Conversely, suppose xˆ satisfies AT Axˆ  AT b.
ˆ is
 Then xˆ satisfies (2), which shows that b  Ax
orthogonal to the rows of AT and hence is orthogonal to
the columns of A.
ˆ is
 Since the columns of A span Col A, the vector b  Ax
orthogonal to all of Col A.
© 2012 Pearson Education, Inc.
Slide 6.5- 7
SOLUTION OF THE GENREAL LEAST-SQUARES
PROBLEM
 Hence the equation
ˆ
b  Axˆ  (b  Ax)
is a decomposition of b into the sum of a vector in Col
A and a vector orthogonal to Col A.
 By the uniqueness of the orthogonal decomposition, Axˆ
must be the orthogonal projection of b onto Col A.
ˆ  bˆ and xˆ is a least-squares solution.
 That is, Ax
© 2012 Pearson Education, Inc.
Slide 6.5- 8
SOLUTION OF THE GENREAL LEAST-SQUARES
PROBLEM
 Example 1: Find a least-squares solution of the
inconsistent system Ax  b for
4 0
 2




A  0 2 ,b  0


 
 1 1
11
 Solution: To use normal equations (3), compute:
4 0
 4 0 1 
17 1
T

A A
0 2 




0
2
1
1
5

 1 1 



© 2012 Pearson Education, Inc.
Slide 6.5- 9
SOLUTION OF THE GENREAL LEAST-SQUARES
PROBLEM
 2
 4 0 1   19 
T
A b
0  



0
2
1

 11 11
 
 Then the equation A Ax  A b becomes
T
T
17 1  x1  19 
 1 5  x   11

 2  
© 2012 Pearson Education, Inc.
Slide 6.5- 10
SOLUTION OF THE GENREAL LEAST-SQUARES
PROBLEM
 Row operations can be used to solve the system on the
previous slide, but since ATA is invertible and 2  2, it
is probably faster to compute
1  5 1
( A A)  
84  1 17 
1
T
and then solve A Ax  A b as
T
T
ˆx  ( AT A) 1 AT b
1  5 1 19  1  84   1
= 
   



84  1 17  11 84 168  2 
© 2012 Pearson Education, Inc.
Slide 6.5- 11
SOLUTION OF THE GENREAL LEAST-SQUARES
PROBLEM
 Theorem 14: Let A be an m  n matrix. The following

statements are logically equivalent:
a. The equation Ax  b has a unique least-squares
m
solution for each b in .
b. The columns of A are linearly independent.
c. The matrix ATA is invertible.
When these statements are true, the least-squares
solution xˆ is given by
T
1 T
----(4)
xˆ  ( A A) A b
When a least-squares solution xˆ is used to produce Axˆ
as an approximation to b, the distance from b to Axˆ is
called the least-squares error of this approximation.
© 2012 Pearson Education, Inc.
Slide 6.5- 12
ALTERNATIVE CALCULATIONS OF LEASTSQUARES SOLUTIONS
 Example 2: Find a least-squares solution of Ax  b for
1 6 
 1
1 2 
 2
,b   
A
1 1
 1
1 7 
 6


 
 Solution: Because the columns a1 and a2 of A are
orthogonal, the orthogonal projection of b onto Col A is
given by
ˆb  b a1 a  b a 2 a  8 a  45 a ----(5)
1
2
1
2
a1 a1
© 2012 Pearson Education, Inc.
a2 a2
4
90
Slide 6.5- 13
ALTERNATIVE CALCULATIONS OF LEASTSQUARES SOLUTIONS
 2   3  1
 2   1 

1


 
 2   1/ 2   5 / 2 
 2  7 / 2  11/ 2 
  
 

ˆ  bˆ .
 Now that bˆ is known, we can solve Ax
 But this is trivial, since we already know weights to
place on the columns of A to produce bˆ .
 It is clear from (5) that
© 2012 Pearson Education, Inc.
 8/ 4   2 
xˆ  



 45 / 90  1/ 2 
Slide 6.5- 14
ALTERNATIVE CALCULATIONS OF LEASTSQUARES SOLUTIONS
 Theorem 15: Given an m  n matrix A with linearly
independent columns, let A  QR be a QR factorization
m
of A. Then, for each b in
, the equation Ax  b
has a unique least-squares solution, given by
1 T
----(6)
xˆ  R Q b
 Proof: Let xˆ  R Q b.
1
 Then
T
Axˆ  QRxˆ  QRR 1QT b  QQT b
© 2012 Pearson Education, Inc.
Slide 6.5- 15
ALTERNATIVE CALCULATIONS OF LEASTSQUARES SOLUTIONS
 The columns of Q form an orthonormal basis for Col A.
 Hence, by Theorem 10, QQTb is the orthogonal
projection bˆ of b onto Col A.
ˆ  bˆ , which shows that xˆ is a least-squares
 Then Ax
solution of Ax  b.
 The uniqueness of xˆ follows from Theorem 14.
© 2012 Pearson Education, Inc.
Slide 6.5- 16