Transcript Lecture 8 Optical Flow, Feature Tracking, Normal Flow

Stanford CS223B Computer Vision, Winter 2006
Lecture 7
Optical Flow
Professor Sebastian Thrun
CAs: Dan Maynes-Aminzade, Mitul Saha, Greg Corrado
Slides by: Gary Bradski, Intel Research and Stanford SAIL
1
2
Optical Flow: Outline
•
•
•
•
•
Examples
Formal definition, 1D case
From 1D to 2D: Aperture Problem
Course motion and pyramids
Flow Segmentation
3
Optical Flow
4
* Picture from Khurram Hassan-Shafique CAP5415 Computer Vision 2003
Optical Flow, Example
5
Another Example in Robotics
2) Use optical flow
for near-far
association
1) Rich local sensor
data determines
drivability
3) Associate known
driveability with
visually similar
regions
6
Optical Flow, Example
Harris Corners
David Stavens, Andrew Lookingbill, David Lieb (CS223B 2004)
7
Optical Flow, Example
Optical flow
David Stavens, Andrew Lookingbill, David Lieb (CS223B 2004)
8
Optical Flow, Example
“Reward”
David Stavens, Andrew Lookingbill, David Lieb (CS223B 2004)
9
Optical Flow, Example
Particle Filter
David Stavens, Andrew Lookingbill, David Lieb (CS223B 2004)
10
Optical Flow, Example
Result
David Stavens, Andrew Lookingbill, David Lieb (CS223B 2004)
11
Conference Paper
David Stavens, Andrew Lookingbill, David Lieb (CS223B 2004)
12
Optical Flow: Outline
•
•
•
•
•
Examples
Formal definition, 1D case
From 1D to 2D: Aperture Problem
Course motion and pyramids
Flow Segmentation
13
Optical Flow
Image tracking
Image sequence
(single camera)
3D computation
Tracked sequence
3D structure
+
3D trajectory
14
What is Optical Flow?

v2
p2
p3

v1
p1

v3
Optical Flow

v4
p4
I (t  1)
I (t ), { pi }
Velocity vectors

{vi }
Common assumption:
The appearance of the image patches do not change (brightness constancy)

I ( pi , t )  I ( pi  vi , t  1)
15
Optical Flow Break Down?
Perhaps an aperture problem discussed later.

16
* From Marc Pollefeys COMP 256 2003
Optical Flow Assumptions:
Brightness Constancy
17
* Slide from Michael Black, CS143 2003
Optical Flow Assumptions:
18
* Slide from Michael Black, CS143 2003
Optical Flow Assumptions:
19
* Slide from Michael Black, CS143 2003
Optical Flow: 1D Case
Brightness Constancy Assumption:
{
f (t )  I ( x(t ), t )  I ( x(t  dt), t  dt)
f ( x )
0
t
I  x  I


x t  t  t
Ix
v
Because no change in brightness with time
0
x(t )
It
It
v 
Ix
20
Tracking in the 1D case:
I ( x, t ) I ( x, t  1)
p

v?
x
21
Tracking in the 1D case:
I ( x, t ) I ( x, t  1)
Temporal derivative
It
p

v
x
Ix
Spatial derivative
I
Ix 
x t
I
It 
t
x p

I
v  t
Ix
Assumptions:
• Brightness constancy
• Small motion
22
Tracking in the 1D case:
Iterating helps refining the velocity vector
I ( x, t ) I ( x, t  1)
Temporal derivative at 2nd iteration
It
p
x
Ix
Can keep the same estimate for spatial derivative


I
v  v previous  t
Ix
Converges in about 5 iterations
23
Algorithm for 1D tracking:
For all pixel of interest p:
 Compute local image derivative at p: I x

 Initialize velocity vector: v  0
 Repeat until convergence:

 Compensate for current velocity vector: I ' ( x, t  1)  I ( x  v , t  1)
 Compute temporal derivative: I t  I ' ( p, t  1)  I ( p, t )


 Update velocity vector: v  v  I t
Ix
Requirements:
 Need access to neighborhood pixels round p to compute I x
 Need access to the second image patch, for velocity compensation:
 The pixel data to be accessed in next image depends on current
velocity estimate (bad?)
 Compensation stage requires a bilinear interpolation (because v is
not integer)
 The image derivative I x needs to be kept in memory throughout the 24
iteration process
Optical Flow: Outline
•
•
•
•
•
Examples
Formal definition, 1D case
From 1D to 2D: Aperture Problem
Course motion and pyramids
Flow Segmentation
25
From 1D to 2D tracking
1D:
2D:
I  x  I
 
x t  t  t
0
x(t )
I  x  I  y  I
 
 
x t  t  y t  t  t
I
I
I
u
v
x t
y t
t
0
x(t )
0
x(t )
Shoot! One equation, two velocity (u,v) unknowns…
26
From 1D to 2D tracking
We get at most “Normal Flow” – with one point we can only detect movement
perpendicular to the brightness gradient. Solution is to take a patch of pixels
Around the pixel of interest.
27
* Slide from Michael Black, CS143 2003
How does this show up visually?
Known as the “Aperture Problem”
28
Aperture Problem Exposed
Motion along just an edge is ambiguous
29
Aperture Problem: Example
30
Aperture Problem in Real Life
31
From 1D to 2D tracking
I ( x, y, t  1)
y

v2
I ( x, t ) I ( x, t  1)

v3

v1

v

v4
x
I ( x, y , t )
x
The Math is very similar:

v  G 1b

It
v 
Ix
 I x2
G
 
windowaround p  I x I y
b
 I x It 
  
windowaround p  I y I t 
Window size here ~ 5x5 or 11x11
IxIy 

I y2 
32
More Detail:
Solving the aperture problem
• How to get more equations for a pixel?
– Basic idea: impose additional constraints
• most common is to assume that the flow field is smooth locally
• one method: pretend the pixel’s neighbors have the same (u,v)
– If we use a 5x5 window, that gives us 25 equations per pixel!
33
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
Lukas-Kanade flow
• Prob: we have more equations than unknowns
• Solution: solve least squares problem
– minimum least squares solution given by solution (in d) of:
– The summations are over all pixels in the K x K window
– This technique was first proposed by Lukas & Kanade (1981)
• described in Trucco & Verri reading
34
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
Conditions for solvability
– Optimal (u, v) satisfies Lucas-Kanade equation
When is This Solvable?
• ATA should be invertible
• ATA should not be too small due to noise
– eigenvalues l1 and l2 of ATA should not be too small
• ATA should be well-conditioned
– l1/ l2 should not be too large (l1 = larger eigenvalue)
35
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
Eigenvectors of ATA
• Suppose (x,y) is on an edge. What is ATA?
– gradients along edge all point the same direction
– gradients away from edge have small magnitude
–
is an eigenvector with eigenvalue
– What’s the other eigenvector of ATA?
• let N be perpendicular to
• N is the second eigenvector with eigenvalue 0
• The eigenvectors of ATA relate to edge direction and magnitude
36
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
Edge
– large gradients, all the same
– large l1, small l2
37
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
Low texture region
– gradients have small magnitude
– small l1, small l2
38
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
High textured region
– gradients are different, large magnitudes
– large l1, large l2
39
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
Observation
• This is a two image problem BUT
– Can measure sensitivity by just looking at one of the
images!
– This tells us which pixels are easy to track, which are
hard
• very useful later on when we do feature tracking...
– Once suggestion: Track Harris Corners!
40
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
Optical Flow, Example
Harris Corners
David Stavens, Andrew Lookingbill, David Lieb (CS223B 2004)
41
Optical Flow, Example
Optical flow
David Stavens, Andrew Lookingbill, David Lieb (CS223B 2004)
42
Optical Flow: Outline
•
•
•
•
•
Examples
Formal definition, 1D case
From 1D to 2D: Aperture Problem
Course motion and pyramids
Flow Segmentation
43
Revisiting the small motion assumption
• Is this motion small enough?
– Probably not—it’s much larger than one pixel (2nd order terms dominate)
– How might we solve this problem?
44
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
Reduce the resolution!
45
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
Coarse-to-fine optical flow
estimation
u=1.25 pixels
u=2.5 pixels
u=5 pixels
image It-1
u=10 pixels
image I
46
Gaussian pyramid of image It-1
Gaussian pyramid of image I
Coarse-to-fine optical flow
estimation
run iterative L-K
warp & upsample
run iterative L-K
.
.
.
image IJt-1
image I
47
Gaussian pyramid of image It-1
Gaussian pyramid of image I
Multi-resolution Lucas Kanade
Algorithm
48
Optical Flow Results
49
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
Optical Flow Results
50
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
Optical Flow: Outline
•
•
•
•
•
Examples
Formal definition, 1D case
From 1D to 2D: Aperture Problem
Course motion and pyramids
Flow Segmentation
51
Problem
• Multiple motion types in the image
– (e.g., moving cars on 280 or El Camino Real)
52
Motion Template Idea
53
Motion Segmentation Algorithm
• Stamp the current motion history template with the
system time and overlay it on top of the others:
54
Motion Segmentation Algorithm
• Measure gradients of the overlaid motion history
templates:
55
Motion Segmentation Algorithm
• Threshold large gradients to get rid of motion template
edges resulting from too large of a time delay:
56
Motion Segmentation Algorithm
• Find boundaries of most recent motions
• “Walk around boundary
• If drop not too high, Flood fill downwards to segment motions
Segmented
Motion
Segmented
Motion
57
Motion Segmentation Algorithm
Actually need a two-pass algorithm for labeling all motion
segments:
1. Fill downwards; At bottom, turn around and fill upwards.
2. Keep the union of these fills as the segmented motion.
58
Motion Template for Motion
Segmentation and Gesture
Overlay silhouettes, take gradient for normal optical
flow. Flood fill to segment motions.
Motion
Segmentation
Motion
Segmentation
Pose
Recognition
Gesture
Recognition
59