Inner Product Spaces
Download
Report
Transcript Inner Product Spaces
Chapter 5
Inner Product Spaces
5.1 Length and Dot Product in Rn
5.2 Inner Product Spaces
5.3 Orthonormal Bases:Gram-Schmidt Process
5.4 Mathematical Models and Least Square Analysis
Elementary Linear Algebra
R. Larsen et al. (6 Edition)
投影片設計編製者
淡江大學 電機系 翁慶昌 教授
5.1 Length and Dot Product in Rn
Length:
The length of a vector v (v1 , v2 ,, vn ) in Rn is given by
|| v || v1 v2 vn
2
2
2
Notes: The length of a vector is also called its norm.
Notes: Properties of length
1
2
3
4
v 0
v 1 v is called a unit vector.
v 0 iff v 0
cv c v
Elementary Linear Algebra: Section 5.1, p.278
2/80
Ex 1:
(a) In R5, the length of v (0 , 2 , 1 , 4 , 2) is given by
||v || 0 2 (2) 2 12 4 2 (2) 2 25 5
(b) In R3 the length of v (
2
2
17
2
,
2
17
,
3
17
) is given by
2
17
2 2 3
|| v ||
1
17
17 17 17
(v is a unit vector)
Elementary Linear Algebra: Section 5.1, p.279
3/80
A standard unit vector in Rn:
e1, e2 ,, en 1,0,,0, 0,1,,0, 0,0,,1
Ex:
the standard unit vector in R2:
i, j 1,0, 0,1
the standard unit vector in R3: i, j, k 1,0,0, 0,1,0, 0,0,1
Notes: (Two nonzero vectors are parallel)
u cv
1 c 0 u and v have the same direction
2 c 0 u and v have the opposite direction
Elementary Linear Algebra: Section 5.1, p.279
4/80
Thm 5.1: (Length of a scalar multiple)
Let v be a vector in Rn and c be a scalar. Then
|| cv || | c | || v ||
Pf:
v (v1 , v2 , , vn )
cv (cv1 , cv2 , , cvn )
|| cv || || ( cv1 , cv2 , , cvn ) ||
(cv1 ) 2 (cv2 ) 2 (cvn ) 2
c 2 (v1 v2 vn )
2
2
| c | v1 v2 vn
2
2
2
2
| c | || v ||
Elementary Linear Algebra: Section 5.1, p.279
5/80
Thm 5.2: (Unit vector in the direction of v)
If v is a nonzero vector in
Rn,
v
then the vector u
| |v | |
has length 1 and has the same direction as v. This vector u
is called the unit vector in the direction of v.
Pf:
v is nonzero v 0
1
u
v
v
1
0
v
(u has the same direction as v)
v
1
||u ||
||v || 1
||v || ||v ||
Elementary Linear Algebra: Section 5.1, p.280
(u has length 1 )
6/80
Notes:
(1) The vector
v
is called the unit vector in the direction of v.
|| v ||
(2) The process of finding the unit vector in the direction of v
is called normalizing the vector v.
Elementary Linear Algebra: Section 5.1, p.280
7/80
Ex 2: (Finding a unit vector)
Find the unit vector in the direction of v (3 , 1 , 2) ,
and verify that this vector has length 1.
Sol:
v (3 , 1 , 2) v 3 1 22 14
2
2
v
(3 , 1 , 2)
1
1
2
3
(3 , 1 , 2)
,
,
2
2
2
||v ||
14
14
14
14
3 (1) 2
2
2
2
14
3 1 2
1
14
14 14 14
v
is a unit vector.
v
Elementary Linear Algebra: Section 5.1, p.280
8/80
Distance between two vectors:
The distance between two vectors u and v in Rn is
d (u , v) | |u v | |
Notes: (Properties of distance)
(1)
d (u , v) 0
(2)
d (u , v) 0 if and only if
(3)
d (u , v) d ( v , u)
Elementary Linear Algebra: Section 5.1, p.282
uv
9/80
Ex 3: (Finding the distance between two vectors)
The distance between u=(0, 2, 2) and v=(2, 0, 1) is
d (u , v) ||u v || || (0 2 , 2 0 , 2 1) ||
(2) 2 2 2 12 3
Elementary Linear Algebra: Section 5.1, p.282
10/80
Dot product in Rn:
The dot product of u (u1 , u 2 , , u n ) and v (v1 , v2 , , vn )
is the scalar quantity
u v u1v1 u 2 v2 u n vn
Ex 4: (Finding the dot product of two vectors)
The dot product of u=(1, 2, 0, -3) and v=(3, -2, 4, 2) is
u v (1)(3) (2)( 2) (0)( 4) (3)( 2) 7
Elementary Linear Algebra: Section 5.1, p.282
11/80
Thm 5.3: (Properties of the dot product)
If u, v, and w are vectors in Rn and c is a scalar,
then the following properties are true.
(1) u v v u
(2) u ( v w) u v u w
(3) c(u v) (cu) v u (cv)
(4) v v ||v ||2
(5) v v 0 , and v v 0 if and only if v 0
Elementary Linear Algebra: Section 5.1, p.283
12/80
Euclidean n-space:
Rn was defined to be the set of all order n-tuples of real
numbers. When Rn is combined with the standard
operations of vector addition, scalar multiplication, vector
length, and the dot product, the resulting vector space is
called Euclidean n-space.
Elementary Linear Algebra: Section 5.1, p.283
13/80
Ex 5: (Finding dot products)
u (2 , 2) , v (5 , 8), w (4 , 3)
(a) u v
(b) (u v)w (c) u (2v) (d) | |w | |2 (e) u ( v 2w)
Sol:
(a ) u v (2)(5) (2)(8) 6
(b) (u v)w 6w 6(4 , 3) (24 , 18)
(c) u (2v) 2(u v) 2(6) 12
(d) || w ||2 w w (4)(4) (3)(3) 25
(e) v 2w (5 (8) , 8 6) (13 , 2)
u ( v 2w) (2)(13) (2)(2) 26 4 22
Elementary Linear Algebra: Section 5.1, p.284
14/80
Ex 6: (Using the properties of the dot product)
Given u u 39 u v 3
v v 79
Find (u 2v) (3u v)
Sol:
(u 2v) (3u v) u (3u v) 2v (3u v)
u (3u) u v (2v) (3u) (2v) v
3(u u) u v 6( v u) 2( v v)
3(u u) 7(u v) 2( v v)
3(39) 7(3) 2(79) 254
Elementary Linear Algebra: Section 5.1, p.284
15/80
Thm 5.4: (The Cauchy - Schwarz inequality)
If u and v are vectors in Rn, then
| u v | | |u | || |v | |
( | u v | denotes the absolute value of u v )
Ex 7: (An example of the Cauchy - Schwarz inequality)
Verify the Cauchy - Schwarz inequality for u=(1, -1, 3)
and v=(2, 0, -1)
Sol: u v 1, u u 11, v v 5
u v 1 1
u v u u v v 11 5 55
uv u v
Elementary Linear Algebra: Section 5.1, p.285-286
16/80
The angle between two vectors in Rn:
uv
cos
, 0
| | u | || | v | |
Opposite
direction
uv 0
cos 1
uv 0
2
cos 0
uv 0
Same
direction
2
cos 0
0
cos 0
2
0
cos 1
Note:
The angle between the zero vector and another vector is
not defined.
Elementary Linear Algebra: Section 5.1, p.286
17/80
Ex 8: (Finding the angle between two vectors)
u (4 , 0 , 2 , 2) v (2 , 0 , 1 , 1)
Sol:
u u u
42 02 22 22
24
v v v 22 0 1 12 6
2
2
u v (4)(2) (0)(0) (2)(1) (2)(1) 12
uv
12
12
cos
1
|| u || || v ||
24 6
144
u and v have opposite directions. (u 2v)
Elementary Linear Algebra: Section 5.1, p.286
18/80
Orthogonal vectors:
Two vectors u and v in Rn are orthogonal if
uv 0
Note:
The vector 0 is said to be orthogonal to every vector.
Elementary Linear Algebra: Section 5.1, p.287
19/80
Ex 10: (Finding orthogonal vectors)
Determine all vectors in Rn that are orthogonal to u=(4, 2).
Sol:
u (4 , 2)
Let v (v1 , v2 )
u v (4 , 2) (v1 , v2 )
4v1 2v2
0
t
, v2 t
v1
2
t
v ,t ,
2
Elementary Linear Algebra: Section 5.1, p.287
4
2
0 1
1
0
2
tR
20/80
Thm 5.5: (The triangle inequality)
If u and v are vectors in Rn, then | |u v | | | |u | | | |v | |
Pf:
||u v ||2 (u v) (u v)
u (u v) v (u v) u u 2(u v) v v
|| u ||2 2(u v) || v ||2
|| u ||2 2 | u v | || v ||2
||u ||2 2 ||u ||||v || ||v ||2
(| |u || ||v || )2
|| u v || || u || || v ||
Note:
Equality occurs in the triangle inequality if and only if
the vectors u and v have the same direction.
Elementary Linear Algebra: Section 5.1, p.288
21/80
Thm 5.6: (The Pythagorean theorem)
If u and v are vectors in Rn, then u and v are orthogonal
if and only if
||u v ||2 ||u ||2 ||v ||2
Elementary Linear Algebra: Section 5.1, p.289
22/80
Dot product and matrix multiplication:
u1
u
u 2
u n
v1
v
v 2
v n
u v u T v [u1 u 2
(A vector u (u1 , u2 , , un ) in Rn
is represented as an n×1 column matrix)
v1
v
u n ] 2 [u1v1 u 2 v 2 u n v n ]
v n
Elementary Linear Algebra: Section 5.1, p.289
23/80
Keywords in Section 5.1:
length: 長度
norm: 範數
unit vector: 單位向量
standard unit vector : 標準單位向量
normalizing: 單範化
distance: 距離
dot product: 點積
Euclidean n-space: 歐基里德n維空間
Cauchy-Schwarz inequality: 科西-舒瓦茲不等式
angle: 夾角
triangle inequality: 三角不等式
Pythagorean theorem: 畢氏定理
24/80
5.2 Inner Product Spaces
Inner product:
Let u, v, and w be vectors in a vector space V, and let c be
any scalar. An inner product on V is a function that associates
a real number <u, v> with each pair of vectors u and v and
satisfies the following axioms.
(1) 〈u , v〉〈v , u〉
(2) 〈u , v w〉〈u , v〉〈u , w〉
(3)
c〈u , v〉〈cu , v〉
(4) 〈v , v〉 0 and 〈v , v〉 0 if and only if v 0
Elementary Linear Algebra: Section 5.2, p.293
25/80
Note:
u v dot product (Euclidean inner product for R n )
u , v general inner product for vector space V
Note:
A vector space V with an inner product is called an inner
product space.
Vector space:
Inner product space:
V ,
V ,
Elementary Linear Algebra: Section 5.2, Addition
,
, , ,
26/80
Ex 1: (The Euclidean inner product for Rn)
Show that the dot product in Rn satisfies the four axioms
of an inner product.
Sol:
u (u1 , u2 ,, un ) , v (v1 , v2 ,, vn )
〈u , v〉 u v u1v1 u2v2 unvn
By Theorem 5.3, this dot product satisfies the required four axioms.
Thus it is an inner product on Rn.
Elementary Linear Algebra: Section 5.2, p.293
27/80
Ex 2: (A different inner product for Rn)
Show that the function defines an inner product on R2,
where u (u1 , u2 ) and v (v1 , v2 ) .
〈u , v〉 u1v1 2u 2 v2
Sol:
(a) 〈u , v〉 u1v1 2u2v2 v1u1 2v2u2 〈v , u〉
(b) w (w1 , w2 )
〈u , v w〉 u1 (v1 w1 ) 2u2 (v2 w2 )
u1v1 u1w1 2u2 v2 2u2 w2
(u1v1 2u2 v2 ) (u1w1 2u2 w2 )
〈u , v〉〈u , w〉
Elementary Linear Algebra: Section 5.2, pp.293-294
28/80
(c) c〈u , v〉 c(u1v1 2u2v2 ) (cu1 )v1 2(cu2 )v2 〈cu , v〉
(d ) 〈v , v〉 v 2v2 0
2
1
2
〈v , v〉 0 v1 2v2 0
2
2
v1 v2 0 (v 0)
Note: (An inner product on Rn)
〈u , v〉 c1u1v1 c2u2v2 cnunvn ,
Elementary Linear Algebra: Section 5.2, pp.293-294
ci 0
29/80
Ex 3: (A function that is not an inner product)
Show that the following function is not an inner product on R3.
〈u v〉 u1v1 2u 2 v 2 u 3 v3
Sol:
Let v (1 , 2 , 1)
Then v , v (1)(1) 2(2)(2) (1)(1) 6 0
Axiom 4 is not satisfied.
Thus this function is not an inner product on R3.
Elementary Linear Algebra: Section 5.2, p.294
30/80
Thm 5.7: (Properties of inner products)
Let u, v, and w be vectors in an inner product space V, and
let c be any real number.
(1) 〈0 , v〉〈v , 0〉 0
(2) 〈u v , w〉〈u , w〉〈v , w〉
(3) 〈u , cv〉 c〈u , v〉
Norm (length) of u:
|| u|| 〈u , u〉
Note:
|| u ||2 〈u , u〉
Elementary Linear Algebra: Section 5.2, p.295
31/80
Distance between u and v:
d (u , v) || u v|| u v, u v
Angle between two nonzero vectors u and v:
〈u , v〉
cos
, 0
|| u || || v ||
Orthogonal: (u v)
u and v are orthogonal if〈u , v〉 0 .
Elementary Linear Algebra: Section 5.2, p.296
32/80
Notes:
(1) If | |v | | 1 , then v is called a unit vector.
(2)
v 1
v 0
Normalizin g
v
v
(the unit vector in the
direction of v)
not a unit vector
Elementary Linear Algebra: Section 5.2, p.296
33/80
Ex 6: (Finding inner product)
〈 p , q〉 a0b0 a1b1 anbn is an inner product
Let p( x) 1 2 x 2 , q( x) 4 2 x x 2 be polynomial s in P2 ( x)
(a) 〈p , q〉 ?
(b) || q || ?
(c) d ( p , q) ?
Sol:
(a) 〈p , q〉 (1)(4) (0)(2) (2)(1) 2
(b) ||q || 〈q , q〉 4 2 (2) 2 12 21
(c) p q 3 2 x 3x 2
d ( p , q) || p q || p q, p q
(3) 2 2 2 (3) 2 22
Elementary Linear Algebra: Section 5.2, p.296
34/80
Properties of norm:
(1) | |u | | 0
(2) | |u | | 0 if and only if u 0
(3) | |cu | | | c | | |u | |
Properties of distance:
(1) d (u , v) 0
(2) d (u , v) 0 if and only if u v
(3) d (u , v) d ( v , u)
Elementary Linear Algebra: Section 5.2, p.299
35/80
Thm 5.8:
Let u and v be vectors in an inner product space V.
(1) Cauchy-Schwarz inequality:
〈
| u , v〉| | |u | || |v | |
Theorem 5.4
(2) Triangle inequality:
| |u v | | | |u | | | |v | |
Theorem 5.5
(3) Pythagorean theorem :
u and v are orthogonal if and only if
||u v ||2 ||u ||2 ||v ||2
Elementary Linear Algebra: Section 5.2, p.299
Theorem 5.6
36/80
Orthogonal projections in inner product spaces:
Let u and v be two vectors in an inner product space V,
such that v 0. Then the orthogonal projection of u
onto v is given by
u , v
projv u
v
v , v
Note:
If v is a init vector, then 〈v , v〉 || v ||2 1.
The formula for the orthogonal projection of u onto v
takes the following simpler form.
projvu u , v v
Elementary Linear Algebra: Section 5.2, p.301
37/80
Ex 10: (Finding an orthogonal projection in R3)
Use the Euclidean inner product in R3 to find the
orthogonal projection of u=(6, 2, 4) onto v=(1, 2, 0).
Sol:
u , v (6)(1) (2)(2) (4)(0) 10
v , v 12 22 02 5
uv
projvu
v 105 (1 , 2 , 0) (2 , 4 , 0)
vv
Note:
u projv u (6, 2, 4) (2, 4, 0) (4, 2, 4) is orthogonaltov (1,2,0).
Elementary Linear Algebra: Section 5.2, p.301
38/80
Thm 5.9: (Orthogonal projection and distance)
Let u and v be two vectors in an inner product space V,
such that v 0 . Then
d (u , projvu) d (u , cv) ,
Elementary Linear Algebra: Section 5.2, p.302
u , v
c
v , v
39/80
Keywords in Section 5.2:
inner product: 內積
inner product space: 內積空間
norm: 範數
distance: 距離
angle: 夾角
orthogonal: 正交
unit vector: 單位向量
normalizing: 單範化
Cauchy – Schwarz inequality: 科西 - 舒瓦茲不等式
triangle inequality: 三角不等式
Pythagorean theorem: 畢氏定理
orthogonal projection: 正交投影
40/80
5.3 Orthonormal Bases: Gram-Schmidt Process
Orthogonal:
A set S of vectors in an inner product space V is called an
orthogonal set if every pair of vectors in the set is orthogonal.
S v1 , v 2 ,, v n V
Orthonormal:
vi , v j 0
i j
An orthogonal set in which each vector is a unit vector is
called orthonormal.
Note:
S v1 , v 2 , , v n V
1
vi , v j
0
i j
i j
If S is a basis, then it is called an orthogonal basis or an
orthonormal basis.
Elementary Linear Algebra: Section 5.3, p.306
41/80
Ex 1: (A nonstandard orthonormal basis for R3)
Show that the following set is an orthonormal basis.
v1
1
1
S
,
, 0 ,
2
2
v2
2
2 2 2
6 , 6 , 3 ,
v3
2 2 1
, ,
3 3 3
Sol:
Show that the three vectors are mutually orthogonal.
v1 v 2 16 16 0 0
v1 v 3
2
3 2
2
3 2
0 0
2
2 2 2
v 2 v3
0
9
9
9
Elementary Linear Algebra: Section 5.3, p.307
42/80
Show that each vector is of length 1.
| |v 1 | | v 1 v 1
12 0 1
1
2
1
2
36
8
9
94
1
9
1
| |v 2 | | v 2 v 2
2
36
| |v 3 | | v 3 v 3
4
9
Thus S is an orthonormal set.
Elementary Linear Algebra: Section 5.3, p.307
43/80
Ex 2: (An orthonormal basis for P3 ( x ) )
In P3 ( x ) , with the inner product
p, q a0b0 a1b1 a2b2
The standard basis B {1, x, x2} is orthonormal.
Sol:
v1 1 0x 0x 2 ,
v 2 0 x 0x2 ,
v3 0 0 x x 2 ,
Then
v1 , v 2 (1)(0) (0)(1) (0)(0) 0,
v1 , v 3 (1)(0) (0)(0) (0)(1) 0,
v 2 , v 3 (0)(0) (1)(0) (0)(1) 0
Elementary Linear Algebra: Section 5.3, p.308
44/80
v1 v1 , v1
v2 v2 , v2
v3 v3 , v3
11 00 00 1,
00 11 00 1,
00 00 11 1
Elementary Linear Algebra: Section 5.3, p.308
45/80
Thm 5.10: (Orthogonal sets are linearly independent)
If S v1 , v 2 ,, v n is an orthogonal set of nonzero vectors
in an inner product space V, then S is linearly independent.
Pf:
S is an orthogonal set of nonzero vectors
i.e.
v i , v j 0 i j and v i , v i 0
Let
c1 v1 c2 v 2 cn v n 0
c1 v1 c2 v 2 cn v n , v i 0, v i 0
i
c1 v1 , vi c2 v 2 , vi ci vi , vi cn v n , vi
ci vi , vi
v i , v i 0 ci 0 i
Elementary Linear Algebra: Section 5.3, p.309
S is linearly independen t.
46/80
Corollary to Thm 5.10:
If V is an inner product space of dimension n, then any
orthogonal set of n nonzero vectors is a basis for V.
Elementary Linear Algebra: Section 5.3, p.310
47/80
Ex 4: (Using orthogonality to test for a basis)
Show that the following set is a basis for R 4 .
v1
v2
v3
v4
S {(2 , 3 , 2 , 2) , (1 , 0 , 0 , 1) , (1 , 0 , 2 , 1) , (1 , 2 , 1 , 1)}
Sol:
v1 , v 2 , v3 , v 4 : nonzero vectors
v1 v 2 2 0 0 2 0
v 2 v 3 1 0 0 1 0
v1 v 3 2 0 4 2 0
v 2 v 4 1 0 0 1 0
v1 v 4 2 6 2 2 0
v3 v 4 1 0 2 1 0
S is orthogonal.
S is a basis for R 4 (by Corollary to Theorem 5.10)
Elementary Linear Algebra: Section 5.3, p.310
48/80
Thm 5.11: (Coordinates relative to an orthonormal basis)
If B {v1 , v 2 , , v n } is an orthonormal basis for an inner
product space V, then the coordinate representation of a vector
w with respect to B is
w w , v1 v1 w , v 2 v 2 w , v n v n
Pf:
B {v1 , v 2 , , v n } is a basis for V
w V
w k1v1 k2 v 2 kn v n (unique representation)
B {v1 , v 2 , , v n } is orthonormal
1
vi , v j
0
i j
i j
Elementary Linear Algebra: Section 5.3, pp.310-311
49/80
〈 w , v i〉 〈 (k1 v1 k 2 v 2 k n v n ) , v i〉
k〈1 v1 , v i〉 k〈i v i , v i〉 k〈n v n , v i〉
ki
i
w w, v1 v1 w, v 2 v 2 w, v n v n
Note:
If B {v1 , v 2 , , v n } is an orthonormal basis for V and w V ,
Then the corresponding coordinate matrix of w relative to B is
w B
w , v1
w , v
2
w , v n
Elementary Linear Algebra: Section 5.3, pp.310-311
50/80
Ex 5: (Representing vectors relative to an orthonormal basis)
Find the coordinates of w = (5, -5, 2) relative to the following
orthonormal basis for R 3 .
B {( 53 , 54 , 0) , ( 54 , 53 , 0) , (0 , 0 , 1)}
Sol:
w, v1 w v1 (5 , 5 , 2) ( 53 , 54 , 0) 1
w, v 2 w v 2 (5, 5 , 2) ( 54 , 53 , 0) 7
w, v 3 w v 3 (5 , 5 , 2) (0 , 0 , 1) 2
1
[ w ]B 7
2
Elementary Linear Algebra: Section 5.3, p.311
51/80
Gram-Schmidt orthonormalization process:
B {u1 , u2 , , un } is a basis for an inner product space V
Let v1 u1
〈
v 2 u 2 projW1 u 2 u 2
〈
〈
v 3 u3 projW2 u3 u3
〈
w1 span({v1})
u 2 , v1〉
v1
v1 , v1〉
w 2 span({v1 , v 2 })
u3 , v1〉
〈 u3 , v 2〉
v1
v2
v1 , v1〉
〈 v 2 , v 2〉
n 1
〈 v n , v i〉
v n u n projWn1 u n u n
vi
i 1〈 v i , v i〉
B' {v1 , v 2 , , v n } is an orthogonal basis.
vn
v1 v 2
B' ' {
,
, ,
} is an orthonormal basis.
v1 v 2
vn
Elementary Linear Algebra: Section 5.3, p.312
52/80
Ex 7: (Applying the Gram-Schmidt orthonormalization process)
Apply the Gram-Schmidt process to the following basis.
u1
B {(1 , 1 , 0) ,
u2
(1 , 2 , 0) ,
u3
(0 , 1 , 2)}
Sol: v1 u1 (1 , 1 , 0)
u 2 v1
3
1 1
v2 u2
v1 (1 , 2 , 0) (1 , 1 , 0) ( , , 0)
v1 v1
2
2 2
u 3 v1
u3 v 2
v3 u3
v1
v2
v1 v1
v2 v2
1
1/ 2 1 1
(0 , 1 , 2) (1 , 1 , 0)
( , , 0) (0 , 0 , 2)
2
1/ 2 2 2
Elementary Linear Algebra: Section 5.3, pp.314-315
53/80
Orthogonal basis
B' {v1 , v 2 , v 3 } {(1, 1, 0), (
1 1
, , 0), (0, 0, 2)}
2 2
Orthonormal basis
v3
v1 v 2
1 1
1 1
B' ' {
,
,
} {(
,
, 0), (
,
, 0), (0, 0,1)}
v1 v 2
v3
2 2
2 2
Elementary Linear Algebra: Section 5.3, pp.314-315
54/80
Ex 10: (Alternative form of Gram-Schmidt orthonormalization process)
Find an orthonormal basis for the solution space of the
homogeneous system of linear equations.
x1 x 2
Sol:
7 x4 0
2 x1 x 2 2 x3 6 x 4 0
1 1 0 7 0
2 1 2 6 0
G. J . E
1 0 2 1 0
0 1 2 8 0
x1 2 s t
2 1
x 2 s 8t
2 8
s t
2
x3 s
1 0
x
t
0 1
4
Elementary Linear Algebra: Section 5.3, p.317
55/80
Thus one basis for the solution space is
B {u1 , u2 } {(2 , 2 , 1 , 0) , (1 , 8 , 0 , 1)}
v1 u1 2, 2,1, 0
u2 , v1
18
2, 2,1, 0
v 2 u2
v1 1, 8, 0,1
v1 , v1
9
3, 4, 2, 1
B' 2,2,1,0 3,4,2,1
(orthogonal basis)
2 2 1 3 4
2
1
B' '
, , ,0 ,
,
,
,
3 3 3 30 30 30 30
(orthonormal basis)
Elementary Linear Algebra: Section 5.3, p.317
56/80
Keywords in Section 5.3:
orthogonal set: 正交集合
orthonormal set: 單範正交集合
orthogonal basis: 正交基底
orthonormal basis: 單範正交基底
linear independent: 線性獨立
Gram-Schmidt Process: Gram-Schmidt過程
57/80
5.4 Mathematical Models and Least Squares Analysis
Orthogonal subspaces:
T hesubspacesW1 and W2 of an inner productspace V are orthogonal
if v1 , v2 0 for all v1 in W1 and all v2 in W2 .
Ex 2: (Orthogonal subspaces)
T hesubspaces
1 1
- 1
W1 span(0, 1 ) and W2 span( 1 )
1 0
1
are orthogonalbecause v1 , v 2 0 for any vectorin W1 and any vectorin W2 is zero.
Elementary Linear Algebra: Section 5.4, p.321
58/80
Orthogonal complement of W:
Let W be a subspace of an inner product space V.
(a) A vector u in V is said to orthogonal to W,
if u is orthogonal to every vector in W.
(b) The set of all vectors in V that are orthogonal to every
vector in W is called the orthogonal complement of W.
W {v V | v , w 0 , w W }
W (read “ W perp”)
Notes:
(1)
0
V
Elementary Linear Algebra: Section 5.4, p.322
(2) V 0
59/80
Notes:
W is a subspace of V
(1) W is a subspace of V
(2) W W 0
(3) (W ) W
Ex:
If V R 2 , W x axis
Then (1) W y - axis is a subspace of R 2
(2) W W (0,0)
(3) (W ) W
Elementary Linear Algebra: Section 5.4, Addition
60/80
Direct sum:
Let W1 and W2 be two subspaces of R n . If each vector x R n
can be uniquely written as a sum of a vector w1 from W1
and a vector w 2 from W2 , x w1 w 2 , then R n is the
direct sum of W1 and W2 , and you can write
.
R n W1 W2
Thm 5.13: (Properties of orthogonal subspaces)
Let W be a subspace of Rn. Then the following properties
are true.
(1) dim(W ) dim(W ) n
(2) R n W W
(3) (W ) W
Elementary Linear Algebra: Section 5.4, p.323
61/80
Thm 5.14: (Projection onto a subspace)
If {u1 , u2 , , ut } is an orthonormal basis for the
subspace S of V, and v V , then
Pf:
projW v v , u1u1 v , u2 u2 v , ut ut
projW v W and{u1 , u2 , , ut } is an orthonorma
l basis for W
projW v projW v, u1 u1 projW v, ut ut
v projW v , u1 u1 v projW v , ut ut
( projW v v projW v )
v, u1 u1 v, ut ut (projW v, ui 0, i)
Elementary Linear Algebra: Section 5.4, p.324
62/80
Ex 5: (Projection onto a subspace)
w1 0, 3, 1, w 2 2, 0, 0, v 1, 1, 3
Find the projection of the vector v onto the subspace W.
Sol:
W span({w1 , w 2 })
w1 , w 2
: an orthogonal basis for W
3
1
w1 w 2
u1 , u 2
,
,
), 1,0,0 :
(0,
10 10
w1 w 2
an orthonormal basis for W
projW v v, u1 u1 v, u2 u2
6
3
1
9 3
(0,
,
) 1,0,0 (1, , )
5 5
10
10 10
Elementary Linear Algebra: Section 5.4, p.325
63/80
Find by the other method:
A w1 , w 2 , b v
Ax b
x ( AT A) 1 AT b
projcs( A)b Ax A( AT A) 1 AT b
Elementary Linear Algebra: Section 5.4, p.325
64/80
Thm 5.15: (Orthogonal projection and distance)
Let W be a subspace of an inner product space V, and v V.
Then for all w W , w projW v
|| v projW v || || v w ||
or || v projW v || min
|| v w ||
wW
( projW v is the best approximation to v from W)
Elementary Linear Algebra: Section 5.4, p.326
65/80
Pf:
v w ( v projW v) (projW v w)
( v projW v) (projW v w)
By the Pythagorean theorem
|| v w||2 || v projW v ||2 || projW v w||2
w projW v projW v w 0
|| v w ||2 || v projW v ||2
|| v projW v |||| v w ||
Elementary Linear Algebra: Section 5.4, p.326
66/80
Notes:
(1) Among all the scalar multiples of a vector u, the
orthogonal projection of v onto u is the one that is
closest to v. (p.302 Thm 5.9)
(2) Among all the vectors in the subspace W, the vector
projW v is the closest vector to v.
Elementary Linear Algebra: Section 5.4, p.325
67/80
Thm 5.16: (Fundamental subspaces of a matrix)
If A is an m×n matrix, then
(1) (CS ( A)) NS ( A )
( NS ( A )) CS ( A)
(2) (CS ( A )) NS ( A)
( NS ( A)) CS ( A )
(3) CS ( A) NS ( AT ) Rm
CS ( A) ( NS ( A)) Rm
(4) CS ( AT ) NS ( A) Rn
CS ( AT ) (CS ( A )) Rn
Elementary Linear Algebra: Section 5.4, p.327
68/80
Ex 6: (Fundamental subspaces)
Find the four fundamental subspaces of the matrix.
1
0
A
0
0
2
0
0
0
0
1
0
0
(reduced row-echelon form)
Sol:
CS ( A) span 1,0,0,0 0,1,0,0 is a subspace of R 4
CS ( A ) RS A span 1,2,0 0,0,1 is a subspace of R 3
NS ( A) span 2,1,0 is a subspace of R 3
Elementary Linear Algebra: Section 5.4, p.326
69/80
1 0 0 0
1 0 0 0
A 2 0 0 0 ~ R 0 1 0 0
0 1 0 0
0 0 0 0
s t
NS ( A ) span 0,0,1,0 0,0,0,1 is a subspace of R 4
Check:
(CS ( A)) NS ( A )
(CS ( A )) NS ( A)
CS ( A) NS ( AT ) R4
CS ( AT ) NS ( A) R3
Elementary Linear Algebra: Section 5.4, p.327
70/80
Ex 3 & Ex 4:
W span({w1 , w 2 })
Let W is a subspace of R4 and w1 (1, 2,1, 0), w 2 (0, 0, 0,1) .
(a) Find a basis for W
(b) Find a basis for the orthogonal complement of W.
Sol:
1
2
A
1
0
w1
0
1
0
0
~R
0
0
1
0
w2
0
1
0
0
Elementary Linear Algebra: Section 5.4, pp.322-323
(reduced row-echelon form)
71/80
(a) W CS A
1,2,1,0, 0,0,0,1
is a basis for W
(b) W CS A NS A
1 2 1 0
A
0
0
0
1
x1 2 s t
2 1
x s
1 0
s t
2
x3 t
0 1
x 0
0 0
4
2,1,0,0 1,0,1,0 is a basis for W
Notes:
(1)
dim(W ) dim(W ) dim( R 4 )
(2) W W R 4
Elementary Linear Algebra: Section 5.4, pp.322-323
72/80
Least squares problem:
Ax b
m n n 1 m 1
(A system of linear equations)
(1) When the system is consistent, we can use the Gaussian
elimination with back-substitution to solve for x
(2) When the system is inconsistent, how to find the “best possible”
solution of the system. That is, the value of x for which the
difference between Ax and b is small.
Elementary Linear Algebra: Section 5.4, p.320
73/80
Least squares solution:
Given a system Ax = b of m linear equations in n unknowns,
the least squares problem is to find a vector x in Rn that
minimizes
Ax b with respect to the Euclidean inner
product on Rn. Such a vector is called a least squares
solution of Ax = b.
Notes:
T heleast square problemis to find a vectorxˆ in R n such that
Axˆ projCS ( A ) b in thecolumn space of A (i.e., Axˆ CS ( A))
is as close as possible to b. T hatis,
b projCS ( A ) b b Axˆ min
b Ax
n
xR
Elementary Linear Algebra: Section 5.4, p.328
74/80
A M mn
x Rn
Ax CS ( A) (CS A is a subspace of R m )
b CS ( A) ( Ax b is an inconsistent system)
Let Axˆ projCS ( A ) b
(b Axˆ ) CS ( A)
b Axˆ (CS ( A)) NS ( A )
A (b Axˆ ) 0
i.e.
A Axˆ A b (the normal system associated with Ax = b)
Elementary Linear Algebra: Section 5.4, pp.327-328
75/80
Note: (Ax = b is an inconsistent system)
The problem of finding the least squares solution of Ax b
is equal to he problem of finding an exact solution of the
associated normal system A Axˆ Ab .
Elementary Linear Algebra: Section 5.4, p.328
76/80
Ex 7: (Solving the normal equations)
Find the least squares solution of the following system
Ax b
1 1
0
1 2 c 0 1
c
1 3 1 3
(this system is inconsistent)
and find the orthogonal projection of b on the column space of A.
Elementary Linear Algebra: Section 5.4, p.328
77/80
Sol:
1 1
1 1 1
3 6
T
A A
1 2
1
2
3
6
14
1 3
0
1 1 1 4
T
A b
1
1 2 3 3 11
the associated normal system
AT Axˆ AT b
3 6 c0 4
6 14 c 11
1
Elementary Linear Algebra: Section 5.4, p.328
78/80
the least squares solution of Ax = b
53
xˆ 3
2
the orthogonal projection of b on the column space of A
1 1 5
61
3 8
projCS ( A ) b Axˆ 1 2 3 6
2
1 3 176
Elementary Linear Algebra: Section 5.4, p.329
79/80
Keywords in Section 5.4:
orthogonal to W: 正交於W
orthogonal complement: 正交補集
direct sum: 直和
projection onto a subspace: 在子空間的投影
fundamental subspaces: 基本子空間
least squares problem: 最小平方問題
normal equations: 一般方程式
80/80