Transcript Linear Algebra Chapter 7 Inner Product Spaces 大葉大學 資訊工程系 黃鈴玲 Inner Product Spaces In this chapter, we extend those concepts of Rn such as: dot product.

Linear Algebra
Chapter 7
Inner Product Spaces
大葉大學 資訊工程系
黃鈴玲
Inner Product Spaces
In this chapter, we extend those concepts of Rn such as:
dot product of two vectors, norm of a vector, angle between
vectors, and distance between points, to general vector space.
This will enable us to talk about the magnitudes of functions
and orthogonal functions. This concepts are used to
approximate functions by polynomials – a technique that is
used to implement functions on calculators and computers.
We will no longer be restricted to Euclidean Geometry,
we will be able to create our own geometries on Rn.
Ch7_2
7.1 Inner Product Spaces
The dot product was a key concept on Rn that led to definitions of norm,
angle, and distance. Our approach will be to generalize the dot product of
Rn to a general vector space with a mathematical structure called an inner
product. This in turn will be used to define norm, angle, and distance for
a general vector space.
Ch7_3
Definition
An inner product on a real spaces V is a function that associates a
number, denoted 〈u, v〉, with each pair of vectors u and v of V. This
function has to satisfy the following conditions for vectors u, v, and w,
and scalar c.
1.〈u, v〉=〈v, u〉 (symmetry axiom)
2.〈u + v, w〉=〈u, w〉+〈v, w〉 (additive axiom)
3.〈cu, v〉= c〈u, v〉 (homogeneity axiom)
4.〈u, u〉 0, and 〈u, u〉= 0 if and only if u = 0
(position definite axiom)
A vector space V on which an inner product is defined is called an
inner product space.
Any function on a vector space that satisfies the axioms of an inner product
defines an inner product on the space.
There can be many inner products on a given vector space.
Ch7_4
Example 1
Let u = (x1, x2), v = (y1, y2), and w = (z1, z2) be arbitrary vectors
in R2. Prove that〈u, v〉, defined as follows, is an inner product
on R2.
〈u, v〉= x1y1 + 4x2y2
Determine the inner product of the vectors (-2, 5), (3, 1) under
this inner product.
Solution
Axiom 1:〈u, v〉= x1y1 + 4x2y2 = y1x1 + 4y2x2 =〈v, u〉
Axiom 2:〈u + v, w〉=〈 (x1, x2) + (y1, y2) , (z1, z2) 〉
=〈 (x1 + y1, x2 + y2), (z1, z2) 〉
= (x1 + y1) z1 + 4(x2 + y2)z2
= x1z1 + 4x2z2 + y1 z1 + 4 y2z2
=〈(x1, x2), (z1, z2)〉+〈(y1, y2), (z1, z2) 〉
=〈u, w〉+〈v, w〉
Ch7_5
Axiom 3:〈cu, v〉= 〈c(x1, x2), (y1, y2)〉
=〈 (cx1, cx2), (y1, y2) 〉
= cx1y1 + 4cx2y2 = c(x1y1 + 4x2y2)
= c〈u, v〉
Axiom 4: 〈u, u〉= 〈(x1, x2), (x1, x2)〉= x12  4 x22  0
2
2
Further, x1  4 x2  0 if and only if x1 = 0 and x2 = 0. That is u = 0.
Thus〈u, u〉 0, and〈u, u〉= 0 if and only if u = 0.
The four inner product axioms are satisfied,
〈u, v〉= x1y1 + 4x2y2 is an inner product on R2.
The inner product of the vectors (-2, 5), (3, 1) is
〈(-2, 5), (3, 1)〉= (-2  3) + 4(5  1) = 14
Ch7_6
Example 2
Consider the vector space M22 of 2  2 matrices. Let u and v
defined as follows be arbitrary 2  2 matrices.
e

a

f
b

, v  
u  



 g
h 
 c
d 
Prove that the following function is an inner product on M22.
〈u, v〉= ae + bf + cg + dh
5 2
2 - 3
and 
Determine the inner product of the matrices 
.

0 1 
9 0
Solution
Axiom 1:〈u, v〉= ae + bf + cg + dh = ea + fb + gc + hd =〈v, u〉
Axiom 3: Let k be a scalar. Then
〈ku, v〉= kae + kbf + kcg + kdh = k(ae + bf + cg + dh) = k〈u, v〉
2 - 3, 5 2  (2  5)  (-3  2)  (0  9)  (1 0)  4
0 1 9 0
Ch7_7
Example 3
Consider the vector space Pn of polynomials of degree  n. Let f
and g be elements of Pn. Prove that the following function
defines an inner product of Pn.
1
f , g   f ( x) g ( x)dx
0
Determine the inner product of polynomials
f(x) = x2 + 2x – 1 and g(x) = 4x + 1
Solution
1
1
Axiom 1: f , g  0 f ( x) g ( x)dx  0 g ( x) f ( x)dx  g , f
1
Axiom 2: f  g , h   [ f ( x)  g ( x)]h( x)dx
0
1
  [ f ( x)h( x)  g ( x)h( x)]dx
0
1
1
0
0
  [ f ( x)h( x)]dx   g ( x)h( x)dx
 f , h  g, h
Ch7_8
We now find the inner product of the functions f(x) = x2 + 2x – 1
and g(x) = 4x + 1
1
x  2 x - 1, 4 x  1   ( x 2  2 x - 1)(4 x  1)dx
2
0
1
  (4 x 3  9 x 2 - 2 x - 1)dx
0
2
Ch7_9
Norm of a Vector
The norm of a vector in Rn can be expressed in terms of the dot
product as follows
( x1 , x2 , , xn )  ( x12    xn2 )
 ( x1 , x2 , , xn )  ( x1 , x2 , , xn )
Generalize this definition:
The norms in general vector space do not necessary have geometric
interpretations, but are often important in numerical work.
Definition
Let V be an inner product space. The norm of a vector v is
denoted ||v|| and it defined by
v  v, v
Ch7_10
Example 4
Consider the vector space Pn of polynomials with inner product
1
f , g   f ( x) g ( x)dx
0
The norm of the function f generated by this inner product is
f 
f, f 
1
2
[
f
(
x
)]
dx

0
Determine the norm of the function f(x) = 5x2 + 1.
Solution Using the above definition of norm, we get
5x  1 
2
1
2
2
[
5
x

1
]
dx

0
1

4
2
[
25
x

10
x
 1]dx


28
0
3
The norm of the function f(x) =
5x2
+ 1 is
28
3
.
Ch7_11
Example 2’ (補充)
Consider the vector space M22 of 2  2 matrices. Let u and v
defined as follows be arbitrary 2  2 matrices.
e

a

f
b

, v  
u  



 g
h 
 c
d 
It is known that the function 〈u, v〉= ae + bf + cg + dh is an
inner product on M22 by Example 2.
The norm of the matrix is u 
u, u  a 2  b 2  c 2  d 2
Ch7_12
Angle between two vectors
The dot product in Rn was used to define angle between vectors.
The angle  between vectors u and v in Rn is defined by
uv
cos 
u v
Definition
Let V be an inner product space. The angle  between two
nonzero vectors u and v in V is given by
u, v
cos 
u v
Ch7_13
Example 5
Consider the inner product space Pn of polynomials with inner
1
product
f , g   f ( x) g ( x)dx
0
The angle between two nonzero functions f and g is given by
1
cos  
f,g
f g


f ( x) g ( x)dx
0
f g
Determine the cosine of the angle between the functions
f(x) = 5x2 and g(x) = 3x
Solution We first compute ||f || and ||g||.
5x 
2
1
 [5x ] dx  5 and 3x 
2 2
0
1
2
[
3
x
]
dx  3

0
Thus
1
cos 

0
f ( x) g ( x)dx
f g
1


0
(5 x 2 )(3x)dx
5 3

15
4
Ch7_14
Example 2” (補充)
Consider the vector space M22 of 2  2 matrices. Let u and v
defined as follows be arbitrary 2  2 matrices.
e

a

f
b

, v  
u  



 g
h 
 c
d 
It is known that the function 〈u, v〉= ae + bf + cg + dh is an
inner product on M22 by Example 2.
The norm of the matrix is u 
u, u  a 2  b 2  c 2  d 2
The angle between u and v is
cos 
u, v
u v

ae  bf  cg  dh
a 2  b2  c2  d 2 e2  f 2  g 2  h2
Ch7_15
Orthogonal Vectors
Def. Let V be an inner product space. Two nonzero vectors u and
v in V are said to be orthogonal if
u, v  0
Example 6
Show that the functions f(x) = 3x – 2 and g(x) = x are orthogonal
in Pn with inner product
1
f , g   f ( x) g ( x)dx.
0
Solution
1
3 x - 2, x   (3 x - 2)( x)dx  [ x 3 - x 2 ]10  0
0
Thus the functions f and g are orthogonal in this inner product
Space.
Ch7_16
Distance
As for norm, the concept of distance will not have direct
geometrical interpretation. It is however, useful in numerical
mathematics to be able to discuss how far apart various
functions are.
Definition
Let V be an inner product space with vector norm defined by
v  v, v
The distance between two vectors (points) u and v is defined
d(u,v) and is defined by
d (u, v)  u - v ( u - v, u - v )
Ch7_17
Example 7
Consider the inner product space Pn of polynomials discussed
earlier. Determine which of the functions g(x) = x2 – 3x + 5 or h(x)
= x2 + 4 is closed to f(x) = x2.
Solution
1
[d ( f , g )]  f - g , f - g  3 x - 5, 3 x - 5   (3 x - 5) 2 dx  13
2
0
1
[d ( f , h)]  f - h, f - h  - 4, - 4   (-4) 2 dx  16
2
0
Thus d ( f , g )  13 and d ( f , h)  4.
The distance between f and h is 4, as we might suspect, g is closer
than h to f.
Ch7_18
Inner Product on Cn
For a complex vector space, the first axiom of inner product is
modified to read u, v  v, u . An inner product can then be
used to define norm, orthogonality, and distance, as far a real
vector space.
Let u = (x1, …, xn) and v = (y1, …, yn) be element of Cn. The most
useful inner product for Cn is
u, v  x1 y1    xn yn
 ※ u  v if u, v  0
※ u  x1 x1   xn xn
※ d (u, v )  u - v
Ch7_19
Example 8
Consider the vectors u = (2 + 3i, -1 + 5i), v = (1 + i, - i) in C2.
Compute
(a)〈u, v〉, and show that u and v are orthogonal.
(b) ||u|| and ||v||
(c) d(u, v)
Solution
(a) u, v  (2  3i)(1 - i )  (-1  5i )(i )  5  i - i - 5  0
thus u and v are orthogonal.
(b) u  (2  3i )(2 - 3i )  (-1  5i )(-1 - 5i )  13  26  39
v  (1  i )(1 - i )  (-i )(i )  3
(c) d (u, v)  u - v  (2  3i, - 1  5i ) - (1  i, - i )
 (1  2i, - 1  6i )
 (1  2u )(1 - 2i )  (-1  6i )(-1 - 6i )  5  37  42
Ch7_20
Homework
Exercise 7.1:
1, 4, 8(a), 9(a), 10, 12, 13, 15, 17(a), 19,
20(a)
Ch7_21
7.2 Non-Euclidean Geometry and
Special Relativity
Different inner products on Rn lead to different measures of
vector norm, angle, and distance – that is, to different geometries.
dot product  Euclidean geometry
other inner products  non-Euclidean geometries
Example
Let u = (x1, x2), v = (y1, y2) be arbitrary vectors in R2. It is proved
that〈u, v〉, defined as follows, is an inner product on R2.
〈u, v〉= x1y1 + 4x2y2
The inner product differs from the dot product in the appearance
of a 4. Consider the vector (0, 1) in this space. The norm of this
vector is
Ch7_22
(0, 1) 
(0, 1), (0, 1)  (0  0)  4(11)  2
The norm of this vector in Euclidean geometry is 1;
in our new geometry, however, the norm is 2.
Figure 7.1
Ch7_23
Consider the vectors (1, 1) and (-4, 1). The inner product of these
vectors is
(1, 1), (-4, 1)  (1 -4)  4(11)  0
Thus these two vectors are orthogonal.
Figure 7.2
Ch7_24
Let us use the definition of distance based on this inner product to
compute the distance between the points(1, 0) and (0, 1). We have
that
d ((1, 0), (0, 1))  (1, 0) - (0, 1)  (1, - 1)

(1, - 1), (1, - 1)  (11)  4(-11)  5
Figure 7.3
Ch7_25
7.4 Least-Squares Curves
To find a polynomial that best fits given data points.
Ax = y :
(1) if n equations, n variables, and A-1 exists
 x = A-1 y
(2) if n equations, m variables with n > m
 overdetermined
How to solve it?
We will introduce a matrix called the pseudoinverse of A,
denoted pinv(A), that leads to a least-squares solution
x = pinv(A)y for an overdetermined system.
Ch7_26
Definition
Let A be a matrix. The matrix (AtA)-1At is called the
pseudoinverse of A and is denoted pinv(A).
Example 1 Find the pseudoinverse of A
 1 2
= - 1 3.
 2 4


1 2 

6 7 
1 -1 2
t


-1 3  
A A

2 3 4 


7
29



2
4


1
1  29 - 7
t
-1
t
( A A)  t adj( A A) 
125 - 7
6
AA
1  29 - 7  1 - 1 2 1 3 - 10 6
t
-1 t
pinv ( A)  ( A A) A 
 




125 - 7
6 2 3 4 25  1
5 2
Solution
Ch7_27
Let Ax = y be a system of n linear equations in m variables with
n > m, where A is of rank m.
Ax=y 
AtAx=Aty

x = (AtA)-1Aty
AtA is invertible
Ax = y
x = pinv(A)y
system
least-squares solution
If the system Ax=y has a unique solution, the least-squares
solution is that unique solution.
If the system is overdetermined, the least-squares solution is the
closest we can get to a true solution.
The system cannot have many solutions.
Ch7_28
Example 2
Find the least-squares solution of the following overdetermined
x y 6
system of equations. Sketch the solution.
-x y 3
Solution
2x  3y  9
1
6 
1 and y  3

9 
3
 
 1 1 6 6

2
- 1 1  
6 11
3 



 2 3
1
1 6 6
t
-1
t
( A A)  t adj( A A)  
30 6 11
AA
1  11 - 6 1 - 1 2 1 5 - 17 4
t
-1 t
pinv ( A)  ( A A) A  
 



30 - 6
6 1 1 3 30 0
12 6
Ch7_29
The matrix of coefficients is
 1
A  - 1
 2

rank(A)=2 
1 -1
At A  
1 1
The least-squares solution is
6 
1 5 - 17 4    12 
pinv ( A)y  
3  


30 0
12 6  3
9
The least-squares solution is the point P( 12 , 3).
Figure 7.9
Ch7_30
Least-Square Curves
Least-squares line or curve minimizes d12  d22   dn2
Figure 7.10
Ch7_31
Example 3
Find the least-squares line for the following data points.
(1, 1), (2, 4), (3, 2), (4, 4)
Solution
Let the equation of the line by y = a + bx. Substituting for these
points into the equation of the line, we get the overdetermined
a b 1
system
a  2b  4
a  3b  2
a  4b  4
We find the least squares solution. The matrix of coefficients A
and column vector y are as follows.
1 1
 1
A  11 23 and y  42
1 4 
4 
 

It can be shown that
1  20 10 0 - 10
t
-1 t
pinv ( A)  ( A A) A  
20 - 6 - 2 2
6
Ch7_32
The least squares solution is
1

1  20 10 0 - 10 4  1 
t
-1 t
  
[( A A) A ]y  

20 - 6 - 2 2
6 2 0.7
4 
Thus a = 1, b = 0.7.
The equation of the least-squares line for this data is
y = 1 + 0.7x
Figure 7.11
Ch7_33
Example 4
Find the least-squares parabola for the following data points.
(1, 7), (2, 2), (3, 1), (4, 3)
Solution
Let the equation of the parabola be y = a + bx + cx2. Substituting
for these points into the equation of the parabola, we get the
abc  7
system
a  2b  4c  2
a  3b  9c  1
a  4b  16c  3
We find the least squares solution. The matrix of coefficients A
and column vector y are as follows.
1 1 1
7 
A  11 23 49  and y  21
1 4 16 
 3



It can be shown that
1  45 -15 - 25 15 
t
-1 t
pinv ( A)  ( A A) A  -31 23 27 -19 
20  5 -5 -5 5
Ch7_34
The least squares solution is
7
 15.25

45
15
25
15
1 

[( At A) -1 At ]y  -31 23 27 -19  21  - 10.05
20  5 -5 -5 5    1.75
 3 

Thus a = 15.25, b = -10.05, c = 1.75.
The equation of the least-squares parabola for these data points is
y = 15.25 – 10.05x + 1.75x2
Figure 7.12
Ch7_35
Theorem 7.1
Let (x1, y1), …, (xn, yn) be a set of n data points. Let y = a0 + … +
amxm be a polynomial of degree m (n > m) that is to be fitted to
these points. Substituting these points into the polynomial leads
to a system Ax = y of n linear equations in the m variables a0, …,
am, where
1 x1  x1m 
 y1 


A  
 
 and y    
 
m
1 xn  xn 
 yn 
The least-squares solution to
this system gives the coefficients
of the least-squares polynomial
Figure 7.13
for these data points.
y’ is the projection of y onto range(A)
Ch7_36
Homework
Exercise 7.4
3, 11, 21
Ch7_37