Transcript Recent progress in optical flow Presented by: Darya Frolova and Denis Simakov.

Recent progress in optical
flow
Presented by: Darya Frolova and Denis Simakov
1
Optical Flow is not in favor
Very popular slide:
Often not using Optical Flow is
stated as one of the main
advantages of a method
Optical Flow methods have a
reputation of either unreliable or
slow
Recent works claim:
Optical Flow can be computed fast and accurately
2
Optical Flow Research: Timeline
Horn&Schunck
Lucas&Kanade
more
methods
many
methods
1981
1992
1998
Seminal papers
Benchmark:
Barron et.al.
Benchmark:
Galvin et.al.
now
A slow and not very consistent improvement in results,
but a lot of useful ingredients were developed
3
In This Lecture
We will describe :
Ingredients for an accurate and robust optical flow
How people combine these ingredients
Fast algorithms
Papers:
•
•
•
•
•
Combining the advantages of local and global optic flow methods (“Lucas/Kanade
meets Horn/Schunck”)
A. Bruhn, J. Weickert, C. Schnörr, 2002 - 2005
High accuracy optical flow estimation based on a theory for warping
T. Brox, A. Bruhn, N. Papenberg, J. Weickert, 2004 - 2005
Real-Time Optic Flow Computation with Variational Methods
A. Bruhn, J. Weickert, C. Feddern, T. Kohlberger, C. Schnörr, 2003 - 2005
Towards ultimate motion estimation: Combining highest accuracy with real-time
performance
A. Bruhn, J. Weickert, 2005
Bilateral filtering-based optical flow estimation with occlusion detection
J.Xiao, H.Cheng, H.Sawhney, C.Rao, M.Isnardi, 2006
4
What is Optical Flow?
…
5
Definitions
The optical flow is a velocity field in the image which
transforms one image into the next image in a
sequence [ Horn&Schunck ]
+
frame #1
=
flow field
frame #2
The motion field … is the projection into the image of
three-dimensional motion vectors [ Horn&Schunck ]
6
Ambiguity of optical flow
flow (2)
flow (1): true motion
Frame 1
7
Applications
optical flow
motion field
•
•
•
•
•
•
•
video compression
3D reconstruction
segmentation
object detection
activity detection
key frame extraction
interpolation in time
We are usually interested in actual motion
8
Outline
Ingredients for an accurate and robust optical flow
Local image constraints on motion
Robust statistics
Spatial coherence
How people combine these ingredients
Fast algorithms
9
Local image constraints
10
Brightness Constancy
I ( x  u, y  v, t  1)  I ( x, y, t )  0
( x  u, y  v, t  1)
( x, y , t )
u
frame t+1
v
frame t
11
Linearized brightness constancy
Deviation from brightness constancy (we want it to be zero)
  I ( x  u, y  v, t  1)  I ( x, y, t )
Complex dependence on u , v
Linearize:
 LIN  I x () u ()  I y () v()  It ()

 I x I y It

u 
v 
1
()  ( x, y, t )
12
Linearized brightness constancy
 LIN  I x I y I t 
u 
v 
1
Let us square the difference:

 I x2

J  I y I x
 I t I x
2
LIN
I x 
 u v 1  I y  I x I y I t
 
 It 
IxI y
I y2
It I y

I x It 

I y It 
I t2 

u 
 v   [u v 1] J
1 
u 
v 
1
J – “motion tensor”, or
“structure tensor”
13
Averaged linearized constraint
2
 LIN
 I x2

J  I y I x
 I t I x
u 
 [u v 1] J v 
1
IxI y
I y2
It I y
I x It 

I y It 
I t2 
J is a function of x, y (a matrix for every point)
Combine over small neighborhoods (more robust to noise):
J
*
=
G  J
14
Method of Lucas&Kanade

2
LIN  GAUSS
u 
 [u v 1] (G  J ) v 
1
Solve independently for each point [ Lucas&Kanade 1981 ]
2
min  LIN
 GAUSS
linear system
 I x2 I x I y I x I t  u 
G  
 v   0
2
 I y I x I y I y I t  1
Can be solved for every point where matrix is not degenerate
15
Lukas&Kanade - Results
Rubik cube
flow field
Hamburg taxi
flow field
16
Brightness is not always constant
Rotating
cylinder
Brightness constancy
does not always hold
I ( x  u, y  v, t  1)  I ( x, y, t )
intensity
intensity derivative
I ( x  u, y  v, t  1)  I ( x, y, t )
Gradient constancy holds
position
position
17
Local constraints - Summary
We have seen
linearized
• brightness constancy

I ( x  u, y  v, t  1)  I ( x, y, t )  0
2
LIN
u 
 [u v 1] J v   0
1
averaged linearized

2
LIN  GAUSS
u 
 [u v 1] (G  J ) v   0
1
• gradient constancy
I ( x  u, y  v, t  1)  I ( x, y, t )  0
18
Local constraints are not
enough!
19
Local constraints work poorly
input video
Optical flow direction using
only local constraints
color encodes
direction as
marked on
the boundary
20
Where local constraints fail
Uniform regions
Motion is not observable in the image (locally)
21
Where local constraints fail
“Aperture problem”
We can estimate only one flow component (normal)
22
Where local constraints fail
Occlusions
We have not seen where some points moved
Occluded regions are marked in red
23
Obtaining support from neighbors
Two main problems with local constraints:
• information about motion is missing in some points
=> need spatial coherency
• constraints do not hold everywhere
=> need methods to combine them robustly
good
missing
wrong
24
Robust combination of
partially reliable data
or: How to hold elections
25
Toy example
Find “best” representative for the set of numbers
x
xi
L2:
E

x  xi
2
 min
i
Influence of xi on E:
Enew  Eold  2xi  x   
proportional to
x  xi
Outliers influence the most
x  mean( xi )
L1:
E
xx
i
 min
i
x i → xi + ∆
Enew  Eold  
equal for all xi
Majority decides
x  median( xi )
26
Elections and robust statistics
many ordinary people
a very rich man
wealth
Oligarchy
Votes proportional to the wealth
like in L2 norm minimization
Democracy
One vote per person
like in L1 norm minimization
27
Combination of two flow constraints
min
  I
warped


 I   I warped  I

video
I warped  I ( x  u, y  v, t  1) ; I  I ( x, y, t )
usual: L2
x2
robust: L1
robust regularized
x2   2
x
ε
 easy to analyze and minimize
– sensitive to outliers
 robust in presence of outliers
– non-smooth: hard to analyze
 smooth: easy to analyze
 robust in presence of outliers
[A. Bruhn, J. Weickert, 2005]
Towards ultimate motion estimation: Combining highest accuracy with real-time performance
28
Spatial Propagation
29
Obtaining support from neighbors
Two main problems with local constraints:
• information about motion is missing in some points
=> need spatial coherency
• constraints do not hold everywhere
=> need methods to combine them robustly
good
missing
wrong
30
Homogeneous propagation
min
 u
video
2
 v
2
u ( x, y, t ) - flow in the x direction
v( x, y, t ) - flow in the y direction

- gradient
This constraint is not correct on motion boundaries
=> over-smoothing of the resulting flow
[Horn&Schunck 1981]
31
Robustness to flow discontinuities
min
 (
| u |  | v | )
2
2
video
x2   2
:
ε
(also known as isotropic flow-driven regularization)
[T. Brox, A. Bruhn, N. Papenberg, J. Weickert, 2004]
High accuracy optical flow estimation based on a theory for warping
32
Selective flow filtering
We want to propagate information
• without crossing image and flow discontinuities
• from “good”
points only (not occluded)
Solution:
use “bilateral” filter in space, intensity, flow;
taking into account occlusions
[J.Xiao, H.Cheng, H.Sawhney, C.Rao, M.Isnardi, 2006]
Bilateral filtering-based optical flow estimation with occlusion detection
33
Bilateral filter
Unilateral (usual)
Bilateral
I
I
x
I
x
Preserves discontinuities!
I
x
[C. Tomasi, R. Manduchi, 1998] Bilateral filtering for gray and color images.
x
34
Using of bilateral filter - Example
cyan rectangle moves to the right
and occludes background region
marked by red
occluded regions
[J.Xiao, H.Cheng, H.Sawhney, C.Rao, M.Isnardi, 2006]
Bilateral filtering-based optical flow estimation with occlusion detection
35
Learning of spatial coherence
• Come to the next lecture …
36
Spatial coherence: Summary
Homogeneous propagation - oversmoothing
Robust statistics with homogeneous
propagation - preserves flow discontinuities
Bilateral filtering - combines information from
regions with similar flow and similar intensities
Handles occlusions
37
Two more useful ingredients
in brief – one slide each
38
2D vs. 3D
Several frames allow more
accurate optical flow estimation
2 frames:
Several
frames:
Multiscale Optical Flow
Linearization: valid only for small flow
I ( x  u, y  v, t  1)  I ( x, y, t )  I x u  I y v  It
pyramid for frame 1
+
pyramid for frame 2
frame 1
warped
?
upsample
+
(other names: “warping”, “coarse-to-fine”, “multiresolution”)
40
Methods
How to make tasty soup with
these ingredients: several recipes
41
Outline
Ingredients for an accurate and robust optical flow
How people combine these ingredients
Lukas & Kanade meet Horn & Schunck
The more ingredients – the better
Bilateral filtering and occlusions
Fast algorithms
42
Combining ingredients
Spatial coherency
Local constraints
–Brightness constancy
–Image gradient constancy
Energy =
–Homogeneous
–Flow-driven
–Bilateral filtering + occlusions
∫ϕ (Data) + ∫ϕ (“Smoothness”)
Combined using robust statistics

Computed coarse-to-fine
Use several frames
43
Combining Local and Global
 I x2

Remember: J   I y I x
 I t I x
Lucas&Kanade
u 
[u v 1] (G  J )  v 

1 
video
IxI y
I y2
It I y
I x It 

I y It 
I t2 
Horn&Schunk
u 
[u v 1] J v  

1
video
u 
[u v 1] (G  J ) v  

1
video


 u

 v
video
u  v
2
2
2
video
Basic “Combining local and global”
[A. Bruhn, J. Weickert, C. Schnörr, 2002 ]
44
2
Sensitivity to noise – quantitative results
frame t+1
Error measure: angle
between true and
computed flow in
(x,y,t) space
frame t
ground truth flow
45
The more ingredients - the better
brightness constancy
spatial coherence
gradient constancy
  I
warped
I
video
    I
warped
 I



   flow 
video
[Bruhn, Weickert, 2005]
Towards ultimate motion estimation: Combining highest accuracy with real-time performance
46
Quantitative results
Angular error
Method
Lucas&Kanade
Average
Standard
Deviation
4.3
(density 35%)
Yosemite sequence with clouds
Horn&Schunk
9.8
16.2
Combining local and
global
4.2
7.7
2.4
6.7
“Towards ultimate …”
Average error decreases, but standard deviation is still high….
47
Influence of each ingredient
For Yosemite sequence with clouds
48
Handling occlusions
bilateral filtering of flow:
preserve intensity and flow
discontinuities;
model occlusions
  I
video
warped
I



 smoothness
video
:
[J.Xiao, H.Cheng, H.Sawhney, C.Rao, M.Isnardi, ECCV 2006]
Bilateral filtering-based optical flow estimation with occlusion detection
49
Qualitative results
50
Quantitative results
Angular error
Method
Lucas&Kanade
Average
Standard
Deviation
4.3
(density 35%)
Yosemite sequence with clouds
Horn&Schunk
9.8
16.2
Combining local and
global
4.2
7.7
“Towards ultimate
…”
2.4
6.7
Bilateral+occlusions
2.6
6.1
51
Outline
Ingredients for an accurate and robust optical flow
How people combine these ingredients
Fast algorithms
Energy functional => discrete equation
Multigrid solver: nearly real-time
52
How to minimize energy
minimize E (u )   F ( x, u, u ' ) dx
Analogy:
minimize f ( x)
Necessary condition:
F  d F  0
u dx u '
Necessary condition
f ' ( x)  0
Euler-Lagrange equation
53
An example
Let us see how to derive discretized equation for 1D Horn & Schuhck
Horn&Schunk
1D version
(simplified):
u 
minimize  [u v 1] J v  
1
video
E (u ) 
 (I
2
x


u  v
2
video
u  2I x I t u  I )  u'
2
2
2
t
2
image
54
Iterative minimization
(simple example)
E (u ) 
 (I
2
x
u  2I x I t u  I )  u'
2
2
t
2
image
Euler-Lagrange
Discretized:
I x2u  I x I t  u' '  0
ai ui  bi 
ui 1  2ui  ui 1
0
2
h
Linear system of
equation for u
uinew  f (u old )
Local iterations:
uinew
old
2
uiold

u
b
h
i
 1 2 i 1 ai h  2
ai h 2  2
55
Life is not a picnic
E (u ) 
 (I
2
x
u  2I x I t u  I )  u'
2
2
t
2
Linear discretized
system
image
I u  I x I t  u' '  0
2
x
E (u ) 
2 2
2
(
I
u

2
I
I
u

I
x t
t )   u ' 
 x
image
I x2u  I x It 
E (u ) 
d
dx
( ' (u' ))  0
Non-linear in u,
non-linear
discretized system
   I ( x  u, t  1)  I ( x, t )    (u' )
image
Even more
complicated
56
Optimization algorithms
Simple iterative minimization
Multigrid: much faster convergence
57
Solving the system
Au  f
How to solve?
Start with some initial guess
u initial
and apply some iterative method
2 components of success:
fast convergence
good initial guess
58
Relaxation smoothes the error
Relaxation schemes have smoothing property:
............
It may take thousands
of iterations to
propagate information
to large distance
Only neighboring
pixels are coupled in
relaxation scheme
59
Relaxation smoothes the error
Examples
1D case:
2D case:
Error of initial guess
Error after 5 relaxation
Error after 15 relaxations
60
Idea: coarser grid
initial grid – fine grid
On a coarser grid low
frequencies become higher
Hence, relaxations can be
more effective
coarse grid – we take every second point
61
Multigrid 2-Level V-Cycle
1. Iterate ⇒ error becomes smooth
2. Transfer error equation to the coarse
level ⇒ low frequencies become high
5. Correct the previous solution
6. Iterate ⇒ remove interpolation
artifacts
4. Transfer error to the fine level
3. Solve for the error on the coarse level
⇒ good error estimation
62
Coarse grid - advantages
Coarsening allows:
make iteration process faster (on the coarse grid we
can effectively minimize the error)
obtain better initial guess
the coarsest grid)
u initial
(solve directly on
go to the
coarsest grid
interpolate u initial
to the finer
grid
solve here the equation
to find
u initial
Au  f
63
Multigrid approach – Full scheme
64
Non-linear: Full Approximation Scheme
u0  currentsolution
A(u)  f
  u  u0
r  f  Au0
Linear
fine level:
coarse level:
Au  f
A  r
A(·)
non-linear
Non-linear
A(u )  f
A(u0   )  f
Difference from the linear case:
Equation for error involves current solution u0:
⇒ Need to transfer current solution to the coarser level
65
Multigrid: Summary
• Used to solve linear Au  f or non-linear A(u)  f
equations
• Method: combine two techniques
– Basic iterative solver: quickly removes high
frequencies of the error
– Coarsening: makes low frequencies high
• Contribution: fast minimization of loosely
coupled equations
66
Fast Optimization: Results
Horn&Schunck
CLG
“Towards
ultimate…”
Time [sec]
frames/sec
Gauss-Seidel
1.15
0.87
Full multigrid
0.016
62.8
Gauss-Seidel
9.52
0.11
FAS-multigrid
0.087
11.5
Gauss-Seidel
34.5
0.03
FAS-multigrid
0.396
2.5
image size: 160 x 120
67
Summary of the Talk
• 25 years of Optical Flow : a lot of useful ingredients were developed:
• local constraints:
• brightness constancy
• gradient constancy
• smoothing techniques:
• homogeneous
• flow-driven (preserving discontinuities)
• bilateral filters
• handling of occlusions
• robust functions
• multiscale
• All ingredients are combined an a global Energy Minimization approach
• This difficult global optimization can be done very fast using Multigrid
68
Thank you!
69