Transcript Lect0LinearVectorSpaces

ECE 6382
Introduction to Linear Vector
Spaces
D. R. Wilton
ECE Dept.
Reference: D.G. Dudley, “Mathematical Foundations for Electromagnetic
Theory,” IEEE Press, 1994.
1
Fields
 A field F is a set of mathematical objects with rules for adding,
subtracting, multiplying, and dividing and with the usual associativity,
commutivity,and distributivity properties.
Fields :
Q - rationals (ratios of integers)
R - real numbers
C - complex numbers
Not a field :
N - natural numbers, {(0),1, 2, }
Z - integers, {, 2, 1,0,1, 2, }
Field properties :
       

 associativity,
              
  
     

 commutativity

1  
existence of an
0  



identity element
  (  )  0
 ( ) 1  1

 existence of inverse

         
distributivity
 Note it is implied that applying the operation rules of addition or multiplication
to elements of the set results in elements of the set. (I.e., the operations
map elements of the set onto itself .)
Linear Vector Spaces
 Given a collection of vector objects, a, b, c,
scalars, ,  , ,
numbers
in a set S and a set of
defined on a field F of real numbers R or complex
, we say S is a linear vector space if the operations of
addition and multiplication are defined with the following rules :
Addition :
i) a  b  c  a   b  c 
ii ) There exists a vector 0 in S such that a  0  0  a  a.
( 0 : a  0  0  a  a, a S)
iii ) For each a in S ,there exists a vector  a in S such that
a   a    a   a  0
(  a  S : a   a    a   a  0, a S)
iv ) a  b  b  a
 - "there exists"
: - "such that"
 - "in" or "a member of"
 - "for all"
 - "therefore"
- "because" or "since"
Linear Vector Spaces, cont’d
Multiplication by a scalar :
v)
   a     a
vi ) 1  a  a
vii )   a  b    a   b
viii )     a   a   a
Operators :
 Valid operations on a vector
or pair of vectors always
yields a vector in the space,
i.e., a, b  S   a   b  S
(  )  (  ),
 ( )
S
a, b, c,
F
 ,  , ,
Linear Vector Spaces, cont’d

Field

F
 a  b
Linear vector
space
a
S
a
0
b
b
A linear vector space enables us to form linear combinations of vector objects.
Linear Vector Space Examples
Example : Ordinary 3 - D vectors R 3
 a   x xˆ   y yˆ   z zˆ or a  ( x ,  y ,  z )
Example : N - dimensional Euclidean (unitary) space, R N (C N )
 a  (1 ,  2 ,  3 ,
,  N ),
b  ( 1 ,  2 ,  3 ,
,  N ),
k , k  F = R ( )
a  b  (1  1 ,  2   2 ,  3   3 , ,  N   N ),
 a  (1 ,  2 ,  3 , ,  N )
Example : C( ,  )
 a  f ( ), b  g ( ) where f ( ), g ( ) are continuous functions on   ( ,  ),
a  b  f ( )  g ( ),
 a   f ( )
Linear Vector Space Examples , cont’d
Example : M MN , matrices of dimension M  N
1N 
1 N 
 11 12
 11 12


 22
2 N 
 22
2 N 
21
21
, b  
 ,
 a














M2
MN 
M2
MN 
 M1
 M1
12  12
1 N   1 N 
 11  11
 11 12
 
 
 22   22
2 N  2 N 
 22
21
21
 ,  a   21
ab  















M1
M2
M2
MN
MN 
 M1
 M 1  M 2
1N 
 2 N 


 MN 
Linear Independence
N vectors, x1 , x 2 , x 3 ,
N
 x
solution of
k
k 1
k
, x N , are independent if the only
 0 is  k  0 for all k  1, 2,
, N .
The vectors are dependent if some  k  0 can be found
N
such that
 x
k 1
k
k
 0.
Example :
 In R 2 , a  (1, 2), b1  (3,6) are dependent since
3a  b1  0. However, a  (1, 2) and b 2  (3,7)
are independent.
Dimensionality
 A linear space S has dimension N  dim S if it posesses a set of N independent
vectors, whereas every set of N  1 vectors is dependent.
Example :
In R 3 , the vectors xˆ , yˆ , zˆ are easily shown to be independent. The dimension must be
three (dim S  3) since any other vector a can be written in terms of these, i.e.,
a  a x xˆ  a y yˆ  a z zˆ (a  a x xˆ  a y yˆ  a z zˆ  0)
 If we find a set of k independent vectors for every k (k  1, 2, ), then S has infinite
dimension (dim S  ).
Example :
The set fk  f k ( )  2 sin k , k  1, 2,
on the following slide.
, N ,  (0,1) is of infinite dimension, as shown
Linear Independence and Dimensionality
Example :
 In C(0,1), the functions fk  f k ( )  2 sin k , k  1, 2,
, N ,  (0,1) are
independent. To show this, examine the expression
N

k 0
k
f k ( )  0
N
(or
 f
k 0
k k
 0).
0, k 
Since  f k ( ) f ( )d    k  
, multiplying both sides by f ( )
1, k 
0
and integrating over the interval  (0,1) yields
1
  0,
 1, 2,
,N
 f k ( ) are independent.
In this case the set  f k ( ) for k  1, 2,
,  is countably infinite, i.e. each
f k ( ) can be put into a one - to - one correspondence with the set of
integers; the associated vector space is thus of infinite dimension
(dim S  ). The set is then said to be independent if every finite subset
is independent. The analysis above shows that is the case here.
Bases
A set x 1 , x 2 ,
x N is a basis for S if the vectors in the set are independent, and if every
vector x  S can be written as a linear combination of the x k ; namely we can write
N
x   k x k .
k 1
This representation is unique, as we can show by contradiction. I.e., we assume
non - uniqueness so that there must exist non - vanishing coefficients  k   k (for
at least one k ) such that
N
x    kx k .
k 1
Subtracting these last two equations yields
 
N
k 1
k
  k  x k  0,
which, in view of the independence of x k , implies
Note: If N is finite and dim S = N,
then “and if” in the first line above
may be replaced by “then”. I.e.,
any N independent vectors form a
basis. Unfortunately, it is not the
case that any infinite set of
independent vectors forms a basis
when dim S = ∞!
 k   k  0  k.
Thus  k   k for all k , contradicting our hypothesis above.
Bases, cont’d
If S is N - dimensional, any linearly independent set x 1 , x 2 ,
x N is a basis.
Indeed, let x  S. Then by definition, the N  1 vectors x, x 1 , x 2 ,
be linearly dependent,
N
 x    k x k  0,   0.
k 1
Solving for x , we have
  k


k 1  
and hence the set x 1 , x 2 ,
x
N

 x k,

,x N
is a basis.
x N must
Bases, cont’d
Example :

The vectors e 1 , e 2 ,
e N where
e 1  (1,0,0,
,0)
e 2  (0,1,0,
,0)
e N  (0,0,0,
,1)
form a basis in R N . First, the e k can be easily be shown to be
linearly independent. Secondly, for any a  R N ,
a  ( 1 ,  2 ,
  1 (1,0,0,
, N )
,0)   2 (0,1,0,
  1e 1   2e 2 
,0) 
e 2  (0,1,0)  yˆ
e 3  (0,0,1)  zˆ
,1)
  N e N I.e., any vector in the space
can be written as a linear combination of e 1 , e 2 ,
e 1  (1,0,0)  xˆ
  N (0,0,0,
e N . In R 3
Bases, cont’d
Example :

R N is N - dimensional.
- First, there exists N independent vectors, e 1 , e 2 ,
eN .
- Second, we show that any set of N  1 vectors is dependent.
Let a1, a 2 ,
, a N , a N 1 be a set of vectors in R N . We must show
there exists  m  , m  1, 2,
N 1

m 1
, N , N  1, not all zero, such that
a  0.
m m
We express the a m's as a linear combination of the bases e k:
N
a m    k( m )e k , m  1, 2,
, N, N  1
k 1
Substituting into the above and reversing the summation order yields
 N 1
(m) 



  m k  e k  0,
k 1  m 1

N
N 1
   m k( m )  0, k  1, 2,
m 1
, N since e k are independent
Bases, cont’d
N 1
   m k( m )  0, k  1, 2,
m 1
1(1)

,N  
 N(1)

1( N 1)    1  0

 

  
( N 1)
 N   N 1  0
The above is an N  N  1 underdetermined system of linear
equations, and hence has at least one non - trivial solution. Therefore
at least one coefficient  m is non - zero. Hence every set a1, a2 ,
, a N , a N 1
is linearly dependent and R N is N - dimensional. (Note that R1 = .)
Inner Product Spaces
Field
 x, y 
 , if S is a complex inner product space
F
 , if S is a real inner product space
S
x
0
F
y
Inner product space
A linear space is a complex inner product space if for every ordered pair
( x, y ) of vectors in S there exists a unique scalar in
such that
a.  x, y    y, x  *
b.  x  y, z    x, z    y , z 
, denoted  x, y ,
The inner product is a generalization
of the dot product of vectors in R3
c.   x, y     x, y ,  
d.  x, x   0, with equality if and only if ("iff") x  0
For a real inner product space, we drop the conjugate (*) in (a) and
require  
in (c). We usually assume a complex inner product space.
Inner Product Spaces, cont’d
Example :
  0, y   0 , which follows from choosing   0 in (c).
Example :
  x,  y    *  x, y 
Proof :  x,  y     y, x  *

   y, x   *
  *  y, x  *
  *  x, y 
Inner Product Spaces, cont’d
Example :
 N - dimensional (unitary) space : a, b  C N
a  (1 ,  2 ,  3 ,
,  N ), b  ( 1 ,  2 ,  3 ,
,  N ),  k ,  k 
N
C is an inner product space if we define  a, b     k  k* ,
N
k 1
since
 N

 N

a.  a, b     k  k*     k*  k  *     k k*  *   b, a  * .
k 1
 k 1

 k 1

b. If c  (1 ,  2 , 3 , ,  N ),
N
N
 a  b, c     k   k  k*
k 1
N
N
k 1
k 1
   k k*    k k*   a, c    b, c 
c.   a, b  
N
N
  *     *    a, b ,
k 1
k
k
k 1
k
k
 .
Inner Product Spaces, cont’d
Example (cont'd) :
d. First, note that  a, a  is non - negative since
N
N
k 1
k 1
 a, a     k k*    k  0.
2
Second, note that if
N
N
k 1
k 1
 a, a     k k*    k  0,   k  0, k
 a  0  (0,0,
2
,0)
Finally, if a  0  (0,0,
,0)
  a, a   0
Inner Product Spaces, cont’d
Example : C( ,  )
 a  f ( ), b  g ( ) where f ( ), g ( ) are continuous functions on   ( ,  ),

Define  a, b    f , g  
 f ( ) g*( ) d
. That this forms a complex inner
product space is proved similarly to the previous case, with integration
replacing summation.
Since the inner product generalizes the notion of a dot product of
vectors in R3, we often read <a,b> as “a dot b” and say that <a,b>
is a “projection of a along b” or vice versa.
The Cauchy-Schwarz-Bunjakowsky (CSB)
Inequality
 x, y    x, x   y , y 
or
 x, y 
2

 x, x   y , y 
To prove, note first that if  x, y   0, the result is automatically true,
so assume y  0   y , y   0 and define

 x, y 
 y, y 
Hence
 x, y 
 x, y   y , x 
2

   y , x    *  x, y     y , y  .
 y, y 
 y, y 
2
The Cauchy-Schwarz-Bunjakowsky (CSB)
Inequality, cont’d
 x, y 
2
   y , x    *  x, y     y , y 
 y, y 
2
Next note that by property (d) and the above result,
0   x   y , x   y    x, x     y , x     y , y    *  x, y 
2
 x, y 
  x, x  
 y, y 
2

 x, y    x, x   y , y 
2
Orthogonality and Orthonormality
A pair of vectors x and y are orthogonal if  x, y   0.
A set of vectors z k , k  1, 2,
 z i , z j   0, i  j .
is an orthogonal set if
(  z i , z i   Ci  0, if z i  0)
It is an orthonormal set if
 z i , z j    ij (   z i , z j   Ci ij , if only orthogonal)
1, i  j
where  ij  
is the Kronecker delta.
 0, i  j
An orthogonal set that does not contain the 0 vector is proper ,
and can be shown to be independent. Indeed, forming
N
 z
k 1
k
k
 0,
and taking the inner product with zi on both sides,
N
   k z k , z i   0, z i   i  z i , z i   0
k 1
  i  0, i  1, 2,
x
, N if N is finite. Otherwise, the
countably infinite set z k , k  1, 2,
is linearly independent.
y
Note that every
vector is orthogonal
to the 0 vector!
Normed Linear Space
x
S is a normed linear space if, for every vector x  S,
there is a unique number x 
xy
such that
a. x  0 , with equality if and only if x  0.
b.  x   x ,   F
c. x1  x 2  x1  x 2
y
x
(triangle inequality)
0
xy

y
S
Though many possible norms exist, we focus on the norm induced by the
inner product ,
x   x, x  .
The Cauchy - Schwarz - Bunjakowsky (CSB) inequality can thus be written as
 x, y   x y .
Note that if one wishes, one can now define the angle  between two vectors x, y as
cos  
 x, y 
1.
x y
Thus, the norm and inner product allow us to generalize the notions of magnitude
and angle between ordinary vectors in R 3.
Normed Linear Space, cont’d
To check that the norm induced by the scalar product satisfies the norm
conditions, namely,
xy
a. x  0 , with equality if and only if x  0,
x
S
b.  x   x ,   F ,
c. x1  x 2  x1  x 2
(triangle inequality) ,
0
y
we first note that since x   x, x  , property (a) is true by inner product property (d) .
Property (b) above is true since  x    x,  x    *  x, x    x .
Finally, for property (c) above, take two vectors x, y  S and note that
x  y   x  y , x  y    x, x    x, y    y , x    y , y 
2
  x, x    x, y    x, y  *  y , y 
 x  2 Re  x, y   y
2

CSB
x  2  x, y   y
2
2
2
(since Re  x, y    x, y  ))

x  2 x y  y  ( x  y )2

xy
2
2
 x  y
Normed Linear Space, cont’d
Example :
 A useful identity is
xy
x  y  x  y   x  y, x  y    x  y, x  y 
2
2
  x, x    y , x    x, y    y , y 
y
xy
  x, x    y , x    x, y    y , y 
2 x 2 y
2
2
(parallelogram law)
Example :
 For a  (1 ,  2 ,
,  N )  C N the norm is
a   a, a  
N
 k
2
(Pythagorean theorem)
k 1
Example :
 For f ( )  C( ,  ) the norm is
f   f, f  


f ( ) d 
2
x
Convergence of a Sequence
Normed spaces allow us to measure the "closeness" between two vectors. Since
x  y  0  x  y,
it is reasonable to say x and y are "close" if
x  y  .
We need this notion of closeness, for example, to discuss the convergence
of a sequence. We say a sequence x k k 1 of vectors x k  S , converges to a

vector x if , given an   0 , there exists a number N such that x  x k  
whenever k  N . We write x k  x or lim x k  x , and note that x  x k  0.
k 
(x k  x if , given an   0,  N : x  x k    k  N )
Continuity of the Inner Product
Example : Continuity of the Inner Product
In approximating vectors, we require the notion of continuity of an inner product,
which is the idea that if a sequence x k k 1 of vectors x k  S converges to x ,then

 x k , h    x, h  (lim  x k , h    x, h )
k 
where h is any vector in S .
Note it is sufficient to show that  x, h    x k , h    x  x k , h   0.
We have, by the CSB inequality, that
 x  xk , h   x  xk
2
2
2
h .
But x  x k  0 since x k  x and hence
 x  x k , h   0.
Note the result is equivalent to
lim  x k , h    lim x k , h    x, h 
k 
k 
(I.e., "The limit of projections of x k is the projection of the limit.")
Convergence in the Cauchy Sense
A sequence x k k 1 of vectors x k  S converges in the Cauchy sense if,

given an   0 there exists a number N such that x m  x n   whenever
min ( m, n )  N , that is,
lim x m  x n  0.
m , n 

If the sequence converges to x , then it also converges in the Cauchy
sense because
xm  xn  xm  x   xn  x   xm  x  xn  x 



 .
2 2
Unfortunately, the reverse may not be true since there is no guarantee

the limit x is in S. What must be done is to enlarge the definition of the set
S to include its limit points to "complete" the set. A normed linear space
S is thus complete if every Cauchy convergence S converges to a
member of S.
Convergence in the Cauchy Sense, cont’d
Example :
= R1 may be constructed by first including
The real number system
all the rationals, i.e. numbers of the form p q where p and q are integers.
As we know, however, there are many irrational numbers (e.g., 2 ,  , e, etc.).
Indeed, any neighborhood of an irrational number contains an infinite number
of rationals that are arbitrarily close to it. A Cauchy sequence of these can be
constructed that converges to the irrational number. Unfortunately, however,
the limit is not in the set of rationals making up the elements of the sequence!
The solution is to append the irrationals to the set of rationals so that every
Cauchy sequence of elements from the new set converges to an element
of the set. This is the sense in which the real numbers are "complete."
xk  
x1  
2
1
1
7
x 11 
22
7

x2 
x
0
1
2
3
4
5
36
7
Convergence in the Cauchy Sense, cont’d
Example :
The normed space C  ,   is incomplete. The following well known
sequence in C  1,1 ,
1    0
0,

f k ( )   k , 0    1 k , k  1, 2,
1, 1 k    1

converges in the Cauchy sense, but not to an element of C  1,1 . To
see this, choose two members of the sequence and form the difference;
for m  k this is
 0,
( m  k ) ,

fm  fk  
 1  k , 1
 0,
1    0
0  1 m
m  1 k
1 k  1
,
m  k  1, 2,
Convergence in the Cauchy Sense, cont’d
1.0
fm
fk

1 1
m k
Thus we have the (admittedly crude, but sufficient) estimate (refer to the figure)
1k
1
fm  fk
2

2
f

f
d


f

f
d




m
k
m
k


2
1

0
1
(= area of
k
)
For k  m we reverse the argument and finally obtain
2
1 1 
f m  f k  max  , 
k m
0,  1    0
so that f m  
, the unit step function. But the step function
1, 0    1
is not continuous at   0, so the limit function is not in the original space of
continuous functions. Clearly, the space of continuous functions is too
restrictive to be complete in the norm induced by the inner product.
Convergence in the Cauchy Sense, cont’d
An alternative sequence
with a different limit at   0 ...
... and still another sequence
with a different limit at   0.
1.0
1.0
fm
fm
0.5
fk

1
1

k
m
0,  1    0
fm  
,
1,
0



1


fk


1 1
m k
0,  1    0

f m  1 2,
 0
1,
0   1

Note, as seen above, several versions of the step function with different values
at   0 can result from different limiting processes!
 Later we will find that these different step functions, which differ in value
only at the point   0, should, in a certain sense, be regarded as the same
function!
Hilbert Spaces
A linear vector space is a Hilbert space H if it is complete in the norm
induced by the inner product.
 R N is a Hilbert space since a sequence of vectors ak  (1k ,  2k ,
converges to a  (1 ,  2 ,
,  Nk )
,  N ) if and only if lim ik  i . Since both
k 
 and i must be real by the completeness of the reals, then a must bein R N .
k
i
 C N is a Hilbert space since the above argument may be applied to the
real and imaginary parts separately.
 R  and C are Hilbert spaces, however, only if, for all the elements ak of
the sequence, the norm induced by the inner product is finite,
ak
2


  ak , ak       *   ik   .
i 1
k
i
k
i
2
i 1
We usually say that the vectors ak in R or Cmust have components
that are square summable.
The set of square - summable sequences in C is called
2
.
Hilbert Spaces, cont’d
 The space of continuous functions C  ,   is incomplete as we have seen from
our example of a sequence of continuous functions leading to the step function.
Completion of this space leads to the space of functions L2  ,   that are square
integrable on the interval  ,   , namely functions f ( ) such that

f
2

  f , f    f ( ) f * ( )d    f ( ) d    .
2


It should be noted, that in order to complete this space, first the definition of
integration must be extended. In particular, we do not simply interpret the integral
in the traditional Riemann sum sense, but rather as a so - called Lebesgue
integral. Secondly, even the notion of equality of two functions must be extended.
We won't discuss these technicalities, which arise only rarely in applications,
except to mention that if two functions differ at only a finite number or countably
infinite number of points, they are treated as equivalent, and their Lebesgue
integrals are equal. Thus, for example, all unit step functions u( ) are treated as
equivalent no matter what (finite) value is assigned to function at the discontinuity,
  0.
The set of square - integrable functions on  ,   is called L2 ,   .
Linear Subspaces
 M, a subset of S, is a linear subspace or linear manifold , provided that  x   y,
 ,   F , is in M whenever x and y are in M .
 M is itself a linear space.
Example
Let x k , k  1, 2,
, N be linearly independent vectors in the Hilbert space . Define the set
of all linear combinations of the N vectors,
N
 k xk .
k 1
Then M is a linear subspace.
Example
In R 2 , let x, y be vectors in the first quadrant. This is not a linear subspace since it is easy
to find  and  such that  x   y is not in the first quadrant.
Linear Subspaces, cont’d
Example
z
x1  3xˆ  2yˆ  zˆ
x 2  0.1xˆ  4yˆ  2zˆ
 x2
y
O
 x1
x
x1
x
x   x1   x 2
x2
• In the example shown, the linear combinations
of x1, x 2 in S = R 3 form a linear subspace M of dim M = 2.
The linear subspace is a (2 - D) plane passing through O in R 3.
• Linear combinations of x1 in S = R 3 form a linear subspace of dim M = 1 that is a (1- D)
line passing through O in R 3.
• In general, linear combinations of a set of independent vectors form a linear subspace
of dim M  dim S that is a hyperplane through O in S.
• The set of vectors that generates a given linear subspace is not unique!
Gram-Schmidt Orthogonalization
Let x k k 1 be a sequence of N independent vectors that generate
N
a linear subspace M  H. The Gram - Schmidt process is a
constructive procedure for generating an orthonormal sequence
e k k 1 from the
N

independent vectors :
z
Let z1  x1 and e1  1 .
z1

Let z 2  x 2   x 2 , e1  e1 and e 2 


z2
.
z2
Let z 3  x 3   x 3 , e1  e1   x 3 , e 2  e 2
z3
and e 3 
.
z3
In general,
z n  x n   x n , e1  e1   x n , e 2  e 2    x n , e n 1  e n 1
zn
and e n 
, n  1, 2, , N .
zn
N
Note that while the sequence e k k 1 is orthonormal, the sequence
N
z k k 1 is orthogonal.
Gram-Schmidt Orthogonalization, cont’d

N
Any linear combination   k e k is also a linear combination of x k ,
N
k 1
  k x k ; i.e., x k k 1 ande k k 1 generate the same linear subspace.
N
N
k 1

The Gram - Schmidt process may terminate early only if z k  0
N
for some k  N ; this can happen if and only if the sequence x k k 1
is not linearly independent.
 The above property can thus be used as a test to determine if a
sequence of vectors contains dependencies or near - dependencies.
In a numerical Gram - Schmidt process, once should become suspicious of
dependencies if z k   or, say, if z k
z k 1   for some k . Determining
if there are dependencies or near - dependencies in data is a common and
important data processing task. Hence it should not be surprising that variants
of the Gram - Schmidt procedure are ubiquitous in numerical processing,
especially in linear analysis!
Gram-Schmidt Orthogonalization, cont’d
Example :
Given the sequence 1, x, x 2 , x 3 ,
 L  1,1 , we use the Gram - Schmidt
process, with inner product  f , g    f ( x ) g ( x )dx, to produce an
2
1
1
orthogonal sequence .




1
Let z1  1  e1 

1
1
 12

2 .
2
(1) dx
1
 12
 12
Let z2  x   x, 2  2
 12
 x  e2 
 12
x

1
1
 3 2 x.
x 2dx
Let z3  x   x , 2  2   x , 3 2 x   x 2  1 3
x2  1 3
 e3 
 45 8  x 2  1 3 .
1
2
2
1  x  1 3 dx
Etc. ...
2
2
2
Gram-Schmidt Orthogonalization, cont’d
The sequence so generated is proportional to the orthogonal
Legendre polynomials,
1
2
e2  3 2 x

P0 ( x )  1

e3  45 8  x 2  1 3

e4 

e5 

e6 

P1 ( x )  x
1
P2 ( x )   3x 2  1
2
1
P3 ( x )   5 x 3  3x 
2
1
P4 ( x )   35 x 4  30 x 2  3
8
1
P5 ( x )   63x 5  70 x 3  15 x 
8
e1 
etc. ...
The Legendre polynomials are orthogonal but not orthonormal ; they
are normalized such that Pk (1)  1 .
Closed Sets

A linear subspace M is closed if it contains all limits of sequences
constructed from members of M.
Example
The space C( ,  ) is a linear subspace since a sum of two continuous
functions is continuous. However, we showed we can construct a sequence
of continuous functions that converges to a discontinuous function. Hence
C( ,  ) is not closed.
Example
If a closed linear subspace is contained in a Hilbert space, M  H, then

M is itself a Hilbert space. Indeed, let xk k 1 be a Cauchy sequence in M.
Then since M  H, xk  x  H. But since M is closed, x  M and therefore
M is a Hilbert space.
Best Approximation in a Hilbert Space
 Let x be a Hilbert space H of dimension N  , and z k k 1 , M  N , be
M
an orthonormal set in H. Consider approximating x by the finite, M term sum
M
x  x M  k zk ,
k 1

xM  M  H
We want to choose the coefficients  k so as to obtain the "best" approximation
to x . Specically, we want to minimize the norm x  x M . We expand x  x M
2
as
follows :
x  xM
2
  x  xM , x  xM 
  x, x    x M , x M    x M , x    x, x M 
M
M
k 1
j 1
M
M
k 1
k 1
  x, x      k z k ,   j z j      k z k , x    x,   k z k 
M
M
 kj
M
M
k 1
k 1
  x, x     k j *  z k , z j     k  z k , x     k *  x, z k 
k 1 j 1
M
M
M
k 1
k 1
  x, x     k    k  x, z k  *    k *  x, z k 
k 1
2
Best Approximation in a Hilbert Space, cont’d
x  xM
2
M
M
M
k 1
k 1
  x, x     k    k  x, z k  *    k *  x, z k 
k 1
2
M
M
k 1
k 1
  x, x     k   x, z k   k   x, z k   *    x, z k 
M
M
2
  x, x     k   x, z k     x, z k  ,
k 1
2
2
k 1
and since all the terms above are positive, the errror is minimized if we choose
 k   x, z k ,
the Fourier coefficients associated with the expansion of x in the orthonormal
set z k k 1 .
M
Best Approximation in a Hilbert Space, cont’d
 The error associated with the M - term expansion may be defined as
εM  x  xM
M
 x    x, z k  z k
k 1
Note that the projection of the error on every member of the set z j  vanishes :
 ε M , z j    x, z j  
  x, z j  
M
  x, z
k 1
k
  zk , z j 
k
  kj   x, z j    x, z j   0 ;
M
  x, z
k 1
hence we have the remarkable result
M
 ε M , x M     k  ε M ,z k   0.
k 1
Best Approximation in a Hilbert Space, cont’d
 In summary, the vector x H has been decomposed into a vector x M  M plus
an error vector ε M , namely,
x  xM  εM
where the error ε M is orthogonal to the approximation vector x M .
 As the figure suggests, minimizing
z
the error is equivalent to making it
orthogonal to the linear subspace
x2
generated by the orthogonal
x
xM
εM
vectors x k .
y
x1
x
linear subspace M
generated by x1 , x 2
Best Approximation in a Hilbert Space, cont’d
Example
The Legendre polynomials satisfy
1

Pk ( x )P ( x )dx 
1
2
k
2k  1
2
2
k ) .
2k  1 2  1
(
Though not orthonormal, renormalization yields the orthonormal functions
2k  1
zk 
Pk ( x ),  z k , z    k
2
1
where  f , g  

f ( x )g ( x )dx.
1
Hence the "best" approximation to x  f , x  ( 1,1) is
x  f ( x) 
M
 k zk  x M
k 1
where we choose
 k   x, z k  
2k  1
2k  1
 f , Pk ( x )  
2
2
1

f ( x ) Pk ( x ) dx,
1
and where "best" implies minimizing the error in the norm induced by the inner product,
M
f ( x)  x M  f ( x)   k zk
k 1
1


1
M
2
f ( x )    k z k dx
k 1
Best Approximation in a Hilbert Space, cont’d
Example
The trigonometric functions sin nxn 1 are orthogonal on x  (0,  ) :
N

 sin mx sin nx dx 
0

2
 mn ,
and may be renormalized to yield orthonormal functions,
2
zn 
sin nx,  z m , z n    mn ,


where  f , g  
 f ( x )g ( x)dx.
0
Hence the "best" approximation to x  f , x  (0,  ) is
N
2 N
x  f ( x)   n zn 
 n sin nx  x N
n 1
where we choose
 n   x, z n  
2


n 1
 f , sin nx  
2


 f ( x)sin nx dx,
0
where again "best" is in terms of minimizing the error in the norm induced by the inner product.
Generally, we do not yet know if x N  x; to answer that question requires the additional notion
of "completeness" of the set z n . But in this case, we recognize this as a Fourier sine series, for

which we do know that x N  x if

0
2
f ( x) x   .
Best Approximation in a Hilbert Space, cont’d
Example
Consider the piecewise constant approximation of y  f ( x )  L2 (0,1) . The sequence
1, n 1  x  n
N
N , n  1, , N
n ( x)  
 0, otherwise
is orthogonal,   m ,  n    mn N .
We want to approximate f ( x ) as
fˆ ( x ) 
n  n ( x )
f ( x)
1.0
N
 n  n ( x)
n
n 1 n
N N
n 1
1
With y n   n ( x ), and  f , g    f ( x ) g ( x )dx , we thus have
0
 f , n 
n 

 n , n 

n
N
n 1
N

f ( x )dx
n
N
n 1
N

dx
1
N

n
N
n 1
N
f ( x )dx
x
1.0
Orthogonal Complement to a Linear Subspace
 A vector e  H is a member of the set M, the orthogonal complement
to M, if it is orthogonal to every vector in M.
 The projection theorem : Let x be any vector in the Hilbert space H, and
let M  H be a closed linear subspace. Then there exists a unique vector
y 0 closest to x in the sense that x  y 0  x  y
for all y in M. The
necessary and sufficient condition that y 0 is the unique minimizing vector
is that x  y 0 is in M .The vector y 0 is called the projection of x onto M;
e  x  y 0 is the projection of x onto M .
The Projection Theorem
z
linear subspace M
(orthogonal to M)
x
x
y0
e  x  y0
y
linear subspace M
The Projection Theorem and Best
Approximation
 
M
 Let y  H , and y j
 H be a linearly independent set of vectors in H . We
j 1
form the sum
yˆ 
M
  jy j
j 1
and seek to approximate y by yˆ with a suitably chosen set of coefficients  j .
Since linear combinations like the above form a linear subspace M, and the subspace
is closed by virtue of the fact that M  H,the limit of sequences in M must be in M,
and hence M is closed. It therefore meets the requirements of the projection
theorem. Since y j  M, j  1, 2, , M , the projection theorem yields
 ( yˆ  y ), y k   0,
k  1, 2, , M
Substituting the series for yˆ and rearranging yields
M
  j  y j , yk 
j 1
  y, y k  ,
k  1, 2, , M
The Projection Theorem and Best
Approximation, cont’d
 The system of equations,
M
  j  y j , yk 
j 1
  y, y k  ,
k  1, 2, , M
may be written in matrix form as
y 2 , y1
y M , y1   1    y , y1  
 y1, y1


   y, y  
y
,
y
y
,
y
y
,
y

1
2
2
2
M
2
2
2 





 



 

y
,
y
y
,
y
y
,
y


y
,
y

2 M
M M  M  
M 
 1 M
where the  j are solutions of the matrix equation and determine yˆ above.
The matrix is the transpose of the so - called Gram matrix, which also
appears in proofs on the independence of the set y j . The result generalizes
the Fourier coefficient result and reduces to it if the y j are orthogonal. I.e.,
the matrix reduces to a diagonal matrix in this case, and the solution is trivial :
j 
 y, y j 
 y j,y j 

 y, y j 
yj
2
The Projection Theorem and Best
Approximation, cont’d
Example
Consider the polynomial approximation of y  f ( x )  L2 (0,1) . The sequence
 n 1 is linearly independent since, by the fundamental theorem of
x n 1
N
algebra, the only solution of the N  1 degree polynomial equation
N
 an x n 1  0,
x  (0,1)
n 1
are its N  1 roots; hence the only solution that holds for all x is an  0.
(Alternatively, set x  0 in the equation and all its non - vanishing derivatives.)
We want to approximate f ( x ) by
fˆ ( x ) 
N
 n x n 1
n 1
The Projection Theorem and Best
Approximation,
cont’d
1
With y n  x n 1, and  f , g    f ( x ) g ( x )dx , we thus have
0
1
1
0
0
1
 y m , y n    x m  n  2dx 
,  y, y m    f ( x ) x m 1dx
m  n 1

1

1
2

1
 N
1
2
1
3
1
N 1

  f ( x ) dx 



1





 2
  f ( x ) xdx 












1   N   f ( x ) x dx 
 

2 N 1 
1
N
1
N 1
1
0
1
0
1
N 1
0
Solution of the matrix equation yields the coefficients of the best approximation.
But notice that, for example, the next - to - last and last columns (rows) approach
one another for large N ,  N11
1
N
1 1
2 N 2 
N
1
N 1
1  ; the resulting
2 N 1 
near - dependencies in row / column entries make it more difficult to obtain accurate
coefficients for large N .
The Projection Theorem and Best
Approximation, cont’d
Example
Consider the piecewise - linear approximation of y  f ( x )  L2 (0,1) . The sequence
 x  xn 1 , x  ( x , x ), n  2, , N
n 1 n
 xn  xn 1
 x x
 n ( x )   x n 1 x , x  ( xn , xn 1 ), n  1, , N  1
 n 1 n
otherwise
 0,

where xn  Nn 11 n  1, 2, , N , is linearly independent
since the only solution of
1.0
N
 an  n ( x )  0
n 1
is an  0, as is easily seen by evaluating it at x  xn .
To approximate f ( x ) by
fˆ ( x ) 
N
 n  n ( x)
n 1
n  n ( x )
f ( x)
x
n
xn 1 xn xn 1
1.0
The Projection Theorem and Best
Approximation, cont’d
1
With y n   n ( x ), and  f , g    f ( x ) g ( x )dx , we thus have
0
1
 ym , yn  
  m ( x) n ( x)dx ,
0
1
 y, y m    f ( x ) m ( x )dx
0
Evaluation of the basis interaction matrix yields the matrix equation
 1 f ( x )  ( x ) dx 
1
0
0
 0

 1   1

1
    0 f ( x )  2 ( x ) dx 
0
2
6



  1

1
2
0
  3     f ( x )  3 ( x ) dx  ,
0
3
N 1 
   0


  


  N  

1
0 0 0
1
 f ( x )  ( x ) dx 
3

N
 0

the solution of which yields the coefficients  n of the best approximation.
1
3
1
6
1
6
2
3
1
6
Operators in Hilbert Space
 A linear operator L is a mapping that assigns to a vector x  S another vector
y  L x  S . We usually write
y  Lx.
The domain of the operator is the set of vectors x for which the mapping is
defined. The range of the operator is the set of vectors y resulting from the
mapping.
 The operator is linear if it operates on vectors x in a linear vector space S
and
L (1x1   2x 2 )  1L x1   2L x 2
 A linear operator with domain DL  H is bounded if there exists a real number
 such that
Lx   x
for all x DL .
Operators in Hilbert Space, cont’d
Example
The matrix
a1N 
a2 N 



a NN 
 a11 a12
a
a22
21

A


aN 1 aN 2
is a linear operator that assigns to x  1  2
y  Ax
 N  the vector
T
 N  . A is a linear operator since
where y   1  2
A(1x1   2x 2 )  1 Ax1   2 Ax 2  y1  y 2 .
T
The concept of inversion can be generalized to linear operators
and is formally written as
x  A1y
exactly as for matrix equations.
Operators in Hilbert Space, cont’d
Example
Let R  be the space of all vectors of finite norm and consisting of a
countably infinite set of real numbers,
a  (1,  2 , 3 , )
where  k  R.. >
The right - shift operator is defined as
AR a  (0, 1,  2 , ).. >
It is easy to show that AR is linear. In addition AR is bounded since
AR a 

2

 k a ,
k 1
and the least upper bound of   1 ..
Operators in Hilbert Space, cont’d
Example
On the complex Hilbert space L2 (0,1) consider the integral equation
1
0 u( x)k ( x, x)dx  f ( x ),
which we can write in operator form as follows :
Lu  f
where L is the linear operator
1
L=  (  )k ( x, x)dx..
0
We will show the operator is bounded if
1 1
0 0
2
k ( x, x) dxdx   ..
This property - - - that the kernel of the integral equation is square integrable - - - is called the Hilbert - Schmidt property , and the operator
it generates is called a Hilbert - Schmidt operator .
Operators in Hilbert Space, cont’d
To show that a Hilbert - Schmidt operator L is bounded if
1 1
0 0
2
k ( x, x) dxdx  ,
we note that
Lu
2
1
  f ( x ) dx
2
0
where, by the CSB inequality (regard x as merely a fixed parameter and the x
integration as an inner product),
2
f ( x) 
Hence
Finally, if
1
0
Lu
2
1
u( x)k ( x, x)dx   u( x) dx
2
0
2
 u
1 1
0 0
1
0
k ( x, x ) dx   u
2 1 1
2
k ( x, x) dx dx .
0 0

2
k ( x, x) dx dx  M 2 , then we have
Lu  M u
2
2
1
0
2
k ( x, x ) dx .
Continuity of Hilbert Operators
A linear operator on a domain DL  H is continuous if,
given an   0, there exists a   0 such that, for every
u 0 , u  DL , L u 0  L u   whenever u 0  u   .
Lu0
DL
u u0


Lu
Continuity of Hilbert Operators, cont’d
 A linear operator L with domain DL  H is continuous if
 

and only if for every sequence u n
 DL converging to
n 1
u 0  DL ,
L u 0  L ( lim u n )  lim L u n .
n 
n 
PROOF :
1. If L is continuous, the operator and limit may be interchanged.
By continuity, given an  we may select a  such that
u 0  u n   , and since u n is a member of a convergent sequence,
u 0  u n   for all n  N ; hence it is also true that L u 0  L u n   ,
for all n  N or, equivalently,
L u 0  lim L u n .
n 
2. If the operator and limit may be interchanged, L is continuous.
We do not need and will not prove this part.
Equivalence of Boundedness and Continuity of
Hilbert Operators
 A linear operator L with domain DL  H is bounded if
and only if it is continuous.
PROOF :
1. If L is bounded, it is also continuous.
Since L is bounded and u 0  DL ,
Lu0  Lun   u0  un
for all u 0  DL . Thus given an   0, we may select any    
and we will have
L u 0  L u n    
when u 0  u n   .
2. If L is continuous, it is also bounded.
We do not need and will not prove this part.
Unbounded Operator Example
Example
Differential operators are generally unbounded, as in this example.
Let L  d dx. If L is bounded, it is also continuous, i.e. for any un  u,
L u  L  lim un   lim L un
 n   n 
cos n x
, n  1, 2,
n
cos n x
so that lim un  lim
 0, and therefore L  lim un   L  0   0.
n
n 
n 
 n  
But lim L un  lim (  sin n x ) does not exist, which contradicts our
We choose un 
n 
n 
assumption that L is bounded.
Matrix Representation of Bounded Hilbert
Operators
Bounded Hilbert operators are uniquely determined by a matrix. Let {zk }k1
be a basis for H . Let L be bounded operator, and let
Lu  f .
Let us expand u in terms of the basis as follows :
N
 k zk .

N 
u  lim
k 1
Since boundedness implies continuity, and by linearity,
N
N
N
k 1
k 1
k 1
 k zk  lim L   k zk  lim   k L zk  f

N 
N 
N 
L u  L lim
Next, project both sides of the latter equality with the basis set,
yielding
N
 k L zk , z j 

N 
<  lim
k 1
  f , z j ,
j  1, 2,
.
Matrix Representation of Bounded Hilbert
Operators, cont’d
By linearity and continuity of the inner product, we have
N
 k  L zk , z j 

N 
lim
k 1
  f , z j ,
j  1, 2,
,
which is essentially an    matrix equation, which we write as
  L z1, z1   L z2 , z1 
  1    f , z1  
 L z , z   L z , z 
      f , z  
1
2
2
2
2 

  2 


   
If this matrix equation can be solved, then the coefficients  k determine
the complete solution according to
N

k 1
k 1
 k zk    k zk

N 
u  lim
The coefficients  k are unique only to the choice of the basis; another
basis choice will yield a different set of coefficients, but the same solution u .
Non-Negative, Positive, and Positive Definite
Operators
 Convergence criteria are well - established for non - negative, positive,
and positive - definite operators that satisfy
 L x, x   0, for all x  DL
(non - negative)
 L x, x   0, for all x  0 in DL
(positive)
2
 L x, x   c 2 x , c  0, and x  DL (positive - definite)
Non - negative, positive, and positive - definite operators are symmetric, i.e.,
 L x, x    x, L x  ,
Indeed, if the inner product is real, as are all the operators defined above, the
operator is symmetric since
 L x, x    L x, x *   x, L x  .
 For positive, and positive - definite operators an energy inner product,
[ x, y ]   L x, y  , with respect to operator L can be defined, and it
becomes a Hilbert space HL on DL . The associated energy
norm is x   L x, x  .
Non-Negative, Positive, and Positive Definite
Operators, cont’d
The following can be established :

For positive - definite operators

1
x .
c
If a sequence un converges to u, un  u, we mean
x 
lim un  u  0 ;
n 
e
if it converges in energy to u,we write un  u, and we mean
lim un  u  0 .
n 
w
If the sequence un  converges weakly to u, un  u, we mean
that for every g  H,
lim  un  u, g   0 .
n 
Non-Negative, Positive, and Positive Definite
Operators, cont’d
We note (but do not prove) the following relationships between
different types of convergence can be established :

If L un is bounded, convergence implies convergence in energy.

Convergence implies weak convergence.

w
Convergence in energy implies L un  f where g HL .
w

If, however, L un is bounded, then L un  f in H.
If is positive - definite, convergence in energy implies convergence.
The Moment Method
 We developed a matrix representation of the operator L for a basis defined
on a Hilbert space. The procedure led to an infinite matrix in an infinite
dimensional space. In reality, one often finds an approximate solution using a
finite number of bases. The method applies to any linear operator, but
convergence proofs may not always be available.
 To solve L u  ff consider
L u  f  0.
Approximate u using a finite set of bases, {uk }kN1, in the domain of the operator :
u  uN 
N
 k uk .
k 1
Now replace u by u N above and make the residual error orthogonal to a set of
testing functions, {wk }kN1 :
 N

 L    k uk   f , wm   0, m  1, 2,


 k 1

,N ,
The Moment Method, cont’d
Rearranging and using linearity and inner product properties, we have
N
 k  L uk , wm  
k 1
 f , wm , m  1, 2,
, N .,
or in matrix form,
 Lmk k    f m 
  L u1, w1   L u2 , w1 
  Lu , w   Lu , w 
2 2
 Lmk    1 2


 L u1, wN   L u2 , wN 

 L u N , w1  
 L u N , w2  
,


 L u N , wN  
  f , w1  



f
,
w

2 
 f m   




f
,
w


N 
There are no convergence proofs for general operators . For many special
cases and for operators appearing in specific applications , convergence
proofs may exist in spaces appropriate to the operator; developing such
convergence proofs is an active area of research in the math community.