Transcript ELE 774 Adaptive Signal Processing

METHOD OF
STEEPEST DESCENT
Week 5
ELE 774 - Adaptive Signal Processing
1
Mean Square Error (Revisited)

For a transversal filter (of length M), the output is written as
and the error term wrt. a certain desired response is
Week 5
ELE 774 - Adaptive Signal Processing
2
Mean Square Error (Revisited)

Following these terms, the MSE criterion is defined as
Quadratic in w !

Substituting e(n) and manupulating the expression, we get
where
Week 5
ELE 774 - Adaptive Signal Processing
3
Mean Square Error (Revisited)

For notational simplicity, express MSE in terms of vector/matrices
where
Week 5
ELE 774 - Adaptive Signal Processing
4
Mean Square Error (Revisited)

We found that the solution (optimum filter coef.s wo) is given by the
Wiener-Hopf eqn.s


Inversion of R can be very costly.
J(w) is quadratic in w → convex in w → for wo,
 Surface has a single minimum and it is global, then

Can we reach to wo, i.e.
Week 5
with a less demanding algorithm?
ELE 774 - Adaptive Signal Processing
5
Basic Idea of the Method of Steepest Descent

Can we find wo in an iterative manner?
Week 5
ELE 774 - Adaptive Signal Processing
6
Basic Idea of the Method of Steepest Descent

Starting from w(0), generate a sequence {w(n)} with the property

Many sequences can be found following different rules.

Method of steepest descent generates points using the gradient
 Gradient of J at point w, i.e.
gives the direction at which
the function increases most.
 Then
gives the direction at which the function
decreases most.
 Release a tiny ball on the surface of J → it follows negative
gradient of the surface.
Week 5
ELE 774 - Adaptive Signal Processing
7
Basic Idea of the Method of Steepest Descent

For notational simplicity, let
, then going in the direction given by the negative gradient

How far should we go in –g → defined by the step size param. μ
 Optimum step size can be obtained by line search - difficult
 Generally a constant step size is taken for simplicity.

Then, at each step improvement in J is (from Taylor series expansion)
Week 5
ELE 774 - Adaptive Signal Processing
8
Application of SD to Wiener Filter

For w(n)

From the theory of Wiener Filter we know that

Then the update eqn. Becomes
which defines a feedback connection.
Week 5
ELE 774 - Adaptive Signal Processing
9
Convergence Analysis

Feedback → may cause stability problems under certain conditions.
 Depends on


The step size, μ
The autocorrelation matrix, R

Does SD converge?
 Under which conditions?
 What is the rate of convergence?

We may use the canonical representation.

Let the weight-error vector be
then the update eqn. becomes
Week 5
ELE 774 - Adaptive Signal Processing
10
Convergence Analysis

Let
be the eigendecomposition of R.

Then

Using QQH=I

Apply the change of coordinates

Then, the update eqn. becomes
Week 5
ELE 774 - Adaptive Signal Processing
11
Convergence Analysis

We know that Λ is diagonal, then the k-th natural mode is
or, with the initial values vk(0), we have

Note the geometric series
Week 5
ELE 774 - Adaptive Signal Processing
12
Convergence Analysis

Obviously for stability
or
or, simply
Why?

Geometric series results in an exponentially decaying curve with
time constant τk, where letting
Week 5
ELE 774 - Adaptive Signal Processing
13
Convergence Analysis

We have
then
but
We know that Q is composed of the eigenvectors of R, then
or


Each filter coefficient decays exponentially.
The overall rate of convergence is limited by the slowest and fastest
modes
Week 5
ELE 774 - Adaptive Signal Processing
14
Convergence Analysis

For small step size

What is v(0)? The initial value v(0) is

For simplicity assume that w(0)=0, then
Week 5
ELE 774 - Adaptive Signal Processing
15
Convergence Analysis

Transient behaviour:
 From the canonical form we know that


then
As long as the upper limit on the step size parameter μ is satisfied,
regardless of the initial point
Week 5
ELE 774 - Adaptive Signal Processing
16
Convergence Analysis

The progress of J(n) for n=0,1,... is called the learning curve.

The learning curve of the steepest-descent algorithm consists of a
sum of exponentials, each of which corresponds to a natural mode
of the problem.
# natural modes = # filter taps

Week 5
ELE 774 - Adaptive Signal Processing
17
Example

A predictor with 2 taps (w1(n) and w2(n)) is used to find the params.
of the AR process

Examine the transient behaviour for
 Fixed step size, varying eigenvalue spread
 Fixed eigenvalue spread, varying step size.
σv2 is adjusted so that σu2=1.

Week 5
ELE 774 - Adaptive Signal Processing
18
Example

The AR process:

Two eigenmodes
Condition number
Week 5
ELE 774 - Adaptive Signal Processing
19
Example (Experiment 1)

Experiment 1: Keep the step size fixed at

Change the eigenvalue spread
Week 5
ELE 774 - Adaptive Signal Processing
20
Example (Experiment 1)
Week 5
ELE 774 - Adaptive Signal Processing
21
Week 5
ELE 774 - Adaptive Signal Processing
22
Example (Experiment 2)

Keep the eigenvalue spread fixed at

Change the step size (μmax=1.1)
Week 5
ELE 774 - Adaptive Signal Processing
23
Week 5
ELE 774 - Adaptive Signal Processing
24
Example (Experiment 2)

Depending on the value of μ, the learning curve can be
 Overdamped, moves smoothly to the min. ((very) small μ)
 Underdamped, oscillates towards the min. (large μ< μmax)
 Critically damped
 Generally rate of convergence is slow for the first two.
Week 5
ELE 774 - Adaptive Signal Processing
25
Observations


SD is a ‘deterministic’ algorithm, i.e. we assume that
 R and p are known exactly.
In practice they can only be estimated
 Sample average?


Can have high computational complexity.
SD is a local search algorithm, but for Wiener filtering,
 the cost surface is convex (quadratic)
 convergence is guaranteed as long as μ< μmax is satisfied.
Week 5
ELE 774 - Adaptive Signal Processing
26
Observations

The origin of SD comes from the Taylor series expansion (as many
other local search optimization algorithms)

Convergence can we very slow.
To speed up the process, second term can also be included as in
the Newton’s Method

Hessian

High computational complexity (inversion), numerical stability
problems.
Week 5
ELE 774 - Adaptive Signal Processing
27