Transcript Computer Vision Spring 2010 15-385,-685 Instructor: S. Narasimhan Wean 5409 T-R 10:30am – 11:50am Lecture #13

Computer Vision
Spring 2010 15-385,-685
Instructor: S. Narasimhan
Wean 5409
T-R 10:30am – 11:50am
Lecture #13
Optical Flow and Motion
Lecture #13
Optical Flow and Motion
• We are interested in finding the movement of
scene objects from time-varying images (videos).
• Lots of uses
–
–
–
–
–
Track object behavior
Correct for camera jitter (stabilization)
Align images (mosaics)
3D shape reconstruction
Special effects
Tracking – Rigid Objects
(Simon Baker, CMU)
Tracking – Non-rigid Objects
(Comaniciu et al, Siemens)
Face Tracking
(Simon Baker et al, CMU)
Applications of Face Tracking
• User Interfaces:
– Mouse Replacement: Head Pose and Gaze Estimation
– Automotive: Windshield Displays, Smart Airbags, Driver
Monitoring
• Face Recognition:
– Pose Normalization
– Model-Based Face Recognition
• Lipreading/Audio-Visual Speech Recognition
• Expression Recognition and Deception Detection
• Rendering and Animation:
– Expression Animations and Transfer
– Low-Bandwidth Video Conferencing
– Audio-Visual Speech Synthesis
(Simon Baker, CMU)
3D Structure from Motion
(David Nister, Kentucky)
3D Structure from Motion
(David Nister, Kentucky)
Behavior Analysis
Query
Result
(Michal Irani, Weizmann)
Motion Field
• Image velocity of a point moving in the scene
v odt
f'
v idt
ri
X
ro
Z
Y
dro
Scene point velocity: v o =
dri dt
Image velocity: v i =
dt
ro
1
ri =
Perspective projection:
f'
ro × Z
Motion field
(
(
dri
ro × Z )v o - (v o × Z )ro
ro ´ v o )´ Z
vi =
= f'
= f'
2
2
dt
(ro × Z)
(ro × Z)
Optical Flow
• Motion of brightness pattern in the image
• Ideally Optical flow = Motion field
Optical Flow
Motion field exists but no optical flow
¹ Motion Field
No motion field but shading changes
Problem Definition: Optical Flow
• How to estimate pixel motion from image H to image I?
– Find pixel correspondences
• Given a pixel in H, look for nearby pixels of the same color in I
• Key assumptions
– color constancy: a point in H looks “the same” in image I
• For grayscale images, this is brightness constancy
– small motion: points do not move very far
Optical Flow Constraint Equation
( x  u t , y  v t )
( x, y )
time t
( x, y )
tim e t  t
Optical Flow: Velocities
(u , v )
Displacement:
(x, y)  (u t , v t )
– Assume brightness of patch remains same in both images:
E ( x  u t , y  v t , t  t )  E ( x, y, t )
– Assume small motion: (Taylor expansion of LHS up to first order)
E
E
E
E ( x, y, t )  x
 y
 t
 E ( x, y, t )
x
y
t
Optical Flow Constraint Equation
E
E
E
x
 y
 t
0
x
y
t
Divide by t and take the limit t  0
dx E dy E E


0
dt x dt y t
u
Constraint Equation
Ex u  Ey v  Et  0
v
NOTE: (u , v ) must lie on a straight line
We can compute
Ex , Ey , Et
using gradient operators!
But, (u,v) cannot be found uniquely with this constraint!
Finding Gradients in X-Y-T
y
time
j+1
k+1
j
k
i
i+1
x
1
Ex 
[(Ei 1, j ,k  Ei 1, j ,k 1  Ei 1, j 1,k  Ei 1, j 1,k 1 )
4 x
 ( Ei , j ,k  Ei , j ,k 1  Ei , j 1,k  Ei , j 1,k 1 )]
Optical Flow Constraint
• Intuitively, what does this constraint mean?
– The component of the flow in the gradient direction is
determined
– The component of the flow parallel to an edge is
unknown
Optical Flow Constraint
Aperture Problem
Aperture Problem
Computing Optical Flow
• Formulate Error in Optical Flow Constraint:
ec 
2
(
E
u

E
v

E
)
 x y t dx dy
image
• We need additional constraints!
• Smoothness Constraint (as in shape from shading and stereo):
Usually motion field varies smoothly in the image.
So, penalize departure from smoothness:
es 
 (u
2
x
 u )  (v  v ) dx dy
2
y
2
x
2
y
image
• Find (u,v) at each image point that MINIMIZES:
e  es  ec
weighting
factor
Discrete Optical Flow Algorithm
Consider image pixel (i,
j)
• Departure from Smoothness Constraint:
1
sij  [(ui 1, j  ui , j ) 2  (ui , j 1  ui , j ) 2 
4
(vi 1, j  vi , j ) 2  (vi , j 1  vi , j ) 2 ]
•Error in Optical Flow constraint equation:
cij  (Eij x uij  Eij y vij  Eijt )2
• We seek the set
{uij } & {vij } that minimize:
e   (sij   cij )
i
j
NOTE: {uij } & {vij }
show up in more than
one term
Discrete Optical Flow Algorithm
• Differentiating
e
w.r.t
vkl & ukl
and setting to zero:
e
 2 (ukl  ukl )  2 ( Exklukl  E ykl vkl  Etkl ) Exkl  0
ukl
e
kl
kl
kl
kl
 2 (vkl  vkl )  2 ( Ex ukl  E y vkl  Et ) E y  0
vkl
•
vkl & ukl
Update Rule:
are averages of
(u , v )
ukln 1  ukln 
v
n 1
kl
v 
n
kl
around pixel
(k , l )
E xkl ukln  E ykl vkln  Etkl
1   [(E xkl ) 2  ( E ykl ) 2 ]
E xkl ukln  E ykl vkln  Etkl
1   [(E )  ( E ) ]
kl 2
x
kl 2
y
E xkl
E
kl
y
Example
Optical Flow Result
Low Texture Region - Bad
– gradients have small magnitude
Edges – so,so (aperture problem)
– large gradients, all the same
High Textured Region - Good
– gradients are different, large magnitudes
Focus of Expansion (FOE)
• Motion of object = - (Motion of Sensor)
• For a given translatory motion and gaze direction, the world
seems to flow out of one point (FOE).
( x, y, z )
f 1
( x' , y ' )
( x0 , y0 , z0 )
After time t, the scene point moves to:
( x, y, z)  ( x0  ut, y0  vt, z0  wt )
x0  ut y0  vt
( x' , y ' )  {
,
}
z0  wt z0  wt
Focus of Expansion (FOE)
( x, y, z )
f 1
( x' , y ' )
( x0 , y0 , z0 )
x0  ut y0  vt
( x' , y ' )  {
,
}
z0  wt z0  wt
• As t varies the image point moves along a straight line in the image
• Focus of Expansion: Lets backtrack time or
u v
( x' , y ' )  { , }
w w
(t  )
Focus of Expansion (FOE) - Example
http://homepages.inf.ed.ac.uk/rbf/BOOKS/BANDB/LIB/bandb7_12.pdf
Revisiting the Small Motion Assumption
• Is this motion small enough?
– Probably not—it’s much larger than one pixel (2nd order terms dominate)
– How might we solve this problem?
Reduce the Resolution!
Coarse-to-fine Optical Flow Estimation
u=1.25 pixels
u=2.5 pixels
u=5 pixels
image
imageHH
Gaussian pyramid of image H
u=10 pixels
image II
image
Gaussian pyramid of image I
Coarse-to-fine Optical Flow Estimation
run iterative OF
upsample
run iterative OF
.
.
.
image
imageHJ
Gaussian pyramid of image H
image II
image
Gaussian pyramid of image I
Image Alignment
• Goal: Estimate single (u,v) translation (transformation) for
entire image
Mosaicing
(Michal Irani, Weizmann)
Mosaicing
(Michal Irani, Weizmann)
Next Class
• Structured Light and Range Imaging
• Reading  Notes