Transcript Optical Flow Methods 2007/8/9

Optical Flow Methods
2007/8/9
Outline





Introduction to 2-D Motion
The Optical Flow Equation
The Solution of Optical Flow Equation
Comparison of different methods
Reference
The 2-D Motion

The projection of 3-D motion into the
image plane.
Z
Y
X
The 2-D Motion(2)

A 2-D displacement field is a collection
of 2-D displacement vectors.
Definition of Optical Flow

Optical flow is a vector field of pixel
velocities based on the observable
variations form the time-varying image
intensity patter.
Difference between Optical
flow and 2-D displacement(1)

There must be sufficient gray-level
variation for the actual motion to be
observable.
Difference between Optical
flow and 2-D displacement(2)

An observable optical flow may not always
correspond to actual motion. For example:
changes in external illumination.
Outline





Introduction to 2-D Motion
The Optical Flow Equation
The Solution of Optical Flow Equation
Comparison of different methods
Reference
The Optical Flow Equation(1)

Let the image brightness at the point (x, y) in
the image plane at time t be denoted by
E ( x, y , t )

The brightness of a particular point in the
pattern is constant, so that dE
dt

0
Using the chain rule for differentiation we see
that, E dx E dy E
x dt

y dt

t
0
The Optical Flow Equation(2)

dx
dy
If we let u  dt and v  dt ,
( Ex , E y )
for the partial derivatives, we have a single
linear equation in two unknowns: u and v.

Writing the equation in the two unknowns
u and v,
E x u  E y v  Et  0
The Optical Flow Equation(3)

Writing the equation in another form,
E

x,
E y  u , v    Et
The component of the movement in the
direction of the brightness gradient equals
(u , v) 
Et
Ex  E y
2
2
The Optical Flow Equation(4)

The velocity has to lie along a line perpendicular
to the brightness gradient vector.
E
x,
E y  u , v    Et
(u , v ) 
( Ex , E y )
Et
Ex  Ey
2
y
2
(u,v)
x
Constraint Line
Outline





Introduction to 2-D Motion
The Optical Flow Equation
The Solution of Optical Flow Equation
Comparison of different methods
Reference
Second-Order
Differential Methods(1)

Based on the conservation of the spatial
image gradient.
d (E ( x, y, t ))
0
dt

The flow field is given by
 E  E 
,

2
vx   x yx 
v    2
2 

E

E
y
 
, 2

 xy y 
2
2
1
 2E 
 tx 


2
  E
 ty 


Second-Order
Differential Methods(2)

The deficiencies:


The constraint does not allow for
some motion such as rotation and
zooming.
Second-order partials cannot always
be estimated with sufficient accuracy.
Block Motion Model (1)
(Lucas and Kanade Method)


Based on the assumption that the
motion vector remains unchanged over
a particular block of pixels.
v( x, y, t )  v(t )
for x,y inside block B
 E
E
E 

E   
 vx (t ) 
 v y (t ) 
x , yB x
y
t 

2
Block Motion Model (2)

Computing the partials of error with
respect to vx and v y , then setting them
equal to zero, we have
 E
E
E  E
 
 
 vx (t ) 
 v y (t ) 
0
x , yB x
y
t  x

 E
E
E  E
 
 
 vx (t ) 
 v y (t ) 
0
x , yB x
y
t  y

Block Motion Model (3)

Solving the equations, we have
E



x , yB x
vx (t ) 
v (t )  
 y    E
 x , yB x

E
E
, 
x x , yB x
E
E
, 
y x , yB y
E 
y 

E 
y 
1
E

 x ,
yB x

  E
 x , yB y
E 
t 

E 
t 
Block Motion Model (4)


It is possible to increase the influence
of the constraints towards the center of
the block by weighted summations.
The accuracy of estimation depends on
the accuracy of the estimated spatial
and temporal partial derivatives.
Horn and Schunck Method(1)

The additional constraint is to
minimize the sum of the squares of
the Laplacians of the optical flow
velocity:
2
2
2
2

u

u
v v
2
2
 u  2  2 and  v  2  2
x
y
x y
Horn and Schunck Method(2)

The minimization of the sum of the errors
in the equation for the rate of changes of
image brightness.
 b  E xu  E y v  Et
and the measure of smoothness in the
velocity flow.
u u v v
c  2  2  2  2
x
y
x y
2
2
2
2
Horn and Schunck Method(3)

Let the total error to be minimized be


      c   dxdy
2


2
2
2
b
The minimization is to be accomplished by
finding suitable values for optical flow
velocity (u ,v).
The solution can be found iteratively.
Horn and Schunck Method:
Directional-Smoothness constraint

The directional smoothness constraint:
 ds  vx  W vx   v y  W v y 
2


T
T
W is a weight matrix depending on the spatial
changes in gray level content of the video.
The directional-smoothness method minimizes
the criterion function:


      ds   dxdy
2
2
2
2
b
Gradient Estimation Using
Finite Differences(1)

To obtain the estimates of the partials,
we can compute the average of the
forward and backward finite differences.
Gradient Estimation Using
Finite Differences(2)

The three partial derivatives of images brightness
at the center of the cube are estimated form the
average of differences along four parallel edges
of the cube.
Gradient Estimation by Local
Polynomial Fitting(1)

An approach to approximate E(x,y,t) locally
by a linear combination of some low-order
polynomials in x, y, and t; that is,
N 1
E ( x, y, t )   aii ( x, y, t )
i 0

Set N equal to 9 and choose the following
basis functions
i ( x, y, t )  1, x, y, t , x 2 , y 2 , xy, xt, yt
Gradient Estimation by Local
Polynomial Fitting(2)

The coefficients are estimated by using the
least squares method. N 1
e 2   ( E ( x, y, t )   aii ( x, y, t )) 2
x

y
t
i 0
The components of the gradient can be
found by differentiation,
E ( x, y , t )
 a1  2a4 x  a6 y  a7 t
x
E ( x, y , t )
 a2  2a5 y  a6 x  a8t
y
E ( x, y , t )
 a3  a7 x  a8 y
t
Estimating the Laplacian of
the Flow Velocities(1)

The approximation takes the following form
 u   (u i , j ,k  ui , j ,k ) and  2v   (v i , j ,k  vi , j ,k )
2

The local averages u and v are defined as:
Estimating the Laplacian of
the Flow Velocities(2)

The Laplacian is estimated by
subtracting the value at a point form a
weighted average of the values at
neighboring points.
Outline





Introduction to 2-D Motion
The Optical Flow Equation
The Solution of Optical Flow Equation
Comparison of different methods
Reference
Comparison of
different methods(1)

Three different method to be compared:



Lucas-Kanade method based on block motion
model. (11x11 blocks with no weighting)
Horn-Schunck method imposing a global
smoothness constraint.(  2  625 , allowed for
20 to 150 iterations)
The directional-smoothness method of
Nagel(  2  25,   5 with 20 iterations)
Comparison of
different methods(2)



These methods have been applied to the 7th
and 8th frames of a video sequence, known
as the “Mobile and Calendar.”
The gradients have been approximated by
average finite differences and polynomial
fitting.
The images are spatially pre-smoothed by a
5x5 Gaussian kernel with the variance 2.5
pixels.
Comparison of
different methods(3)


PSNR  10 log 10
255  255
2
E8 ( x, y)  E7 ( x  d x ( x, y), y  d y ( x, y))


Comparison of the differential methods.
Method
Frame-Difference
Lucas-Kanade
Horn-Schunck
Nagel
PSNR(dB)
Entropy(bits)
Polynomial
Difference
Polynomial
Difference
23.45
30.89
28.14
29.08
32.09
30.71
31.84
6.44
4.22
5.83
6.82
5.04
5.95
Outline





Introduction to 2-D Motion
The Optical Flow Equation
The Solution of Optical Flow Equation
Comparison of different methods
Reference
Reference



A. M. Tekalp, Digital Video Processing.
Englewood Cliffs, NJ: Prentice-Hall, 1995.
Horn, B.K.P. and Schunck, B.G. Determining
optical flow:A retrospective, Artificial
Intelligence, vol. 17, 1981, pp.185-203.
J.L. Barron, D.J. Fleet, and S.S. Beauchemin,
“Performance of Optical Flow Techniques,” in
International Journal of Computer Vision,
February 1994, vol. 12(1), pp. 43-77.