Motion Estimation I

What affects the induced image motion?

• Camera motion • Object motion • Scene structure

Example Flow Fields

• This lesson – estimation of general flow-fields • Next lesson – constrained by global parametric transformations

The Aperture Problem

So how much information is there locally…?

The Aperture Problem

Not enough info in local regions Copyright, 1996 © Dale Carnegie & Associates, Inc.

The Aperture Problem

Not enough info in local regions Copyright, 1996 © Dale Carnegie & Associates, Inc.

The Aperture Problem

Copyright, 1996 © Dale Carnegie & Associates, Inc.

The Aperture Problem Information is propagated from regions with high certainty (e.g., corners) to regions with low certainty.

Copyright, 1996 © Dale Carnegie & Associates, Inc.

Such info propagation can cause optical illusions… Illusory corners

•

Direct (intensity-based) Methods 1. Gradient-based (differential) methods

(Horn &Schunk, Lucase & Kanade)

2. Region-based methods

(Correlation, SSD, Normalized correlation) •

Feature-based Methods

Brightness Constancy Assumption

Image

I

(taken at time

t+1

) Image

J

(taken at time

t

) 

x



u

,

y



v



J



I



x



u

,

y



v



The Brightness Constancy Constraint

Brightness Constancy Equation:

J

(

x

,

y

) 

I

(

x



u

(

x

,

y

) ,

y



v

(

x

,

y

) ) Linearizing (assuming small (u,v) ):

J

(

x

,

y

) 

I

(

x

,

y

) 

I x

(

x

,

y

) 

u

(

x

,

y

) 

I y

(

x

,

y

) 

v

(

x

,

y

) 0 

I x

(

x

,

y

) 

u

(

x

,

y

) 

I y

(

x

,

y

) 

v

(

x

,

y

) 

I

(

x

,

y

) 

J

(

x

,

y

) 

I x

(

x

,

y

) 

u

(

x

,

y

) 

I y

(

x

,

y

) 

v

(

x

,

y

) 

I t

(

x

,

y

)  

I

(

x

,

y

)

T

  

u v

( (

x x

, ,

y y

) )   

I t

(

x

,

y

)

Observations:

* One equation, 2 unknowns * A line constraint in (u,v) space.

* Can recover “Normal-Flow” = 𝐼 𝑡 𝛻𝐼 (the component of the flow in the gradient direction)

Need additional constraints…

Horn and Schunk (1981)

Add global smoothness term

E

 (

x

 ) ,

y



I x u



I y v

 Error in brightness constancy equation

I t

 2    (

x

 ,

y

) 

u x

2 

u

2

y

 

v x

2 

v y

2  Smoothness error Minimize:

E c

 

E s

Solve by using calculus of variations

Horn and Schunk (1981)

Inherent problems: * Smoothness assumption wrong at motion/depth discontinuities  over-smoothing of the flow field.

* How is Lambda determined…?

Lucas-Kanade (1984)

Assume a single displacement (u,v) for all pixels within a small window (e.g., 5x5) Geometrically -- Intersection of multiple line constraints Minimize E(u,v): Algebraically --

E

(

u

,

v

)  (

x

,

y

)  

I



W indow x

(

x

,

y

)

u



I y

(

x

,

y

)

v



I t

 2

Lucas-Kanade (1984)

Minimize E(u,v):

E

(

u

,

v

)  (

x

,

y

)  

I



W indow x

(

x

,

y

)

u



I y

(

x

,

y

)

v



I t

 2

Differentiating w.r.t u and v and equating to 0:

    

I x I I

2

x y

 

I I x y I

2

y

    

u v

     

I I x y I I t t

  

I



I T

 

U

   

I I t

Solve for (u,v) [ Repeat this process for each and every pixel in the image ]

Singularites

    

I I x I x

2

y

 

I I x I

2

y y

   Where in the image will this matrix be invertible and where not…?



Homework

Linearization approximation



iterate & warp

estimate update

x

0 Initial guess: Estimate:

x

Linearization approximation



iterate & warp

estimate update

x

0 Initial guess: Estimate:

x

Linearization approximation



iterate & warp

estimate update

x

0

x

Linearization approximation



iterate & warp

x

0

x

Revisiting the small motion assumption

Is this motion small enough?

Probably not—it’s much larger than one pixel (2 nd terms dominate) How might we solve this problem?

order

Coarse-to-Fine Estimation

Advantages: (i) Larger displacements. (ii) Speedup. (iii) Information from multiple window sizes.

iterate 

u u=1.25 pixels

I x



u



I y



v



I t

+  

u

0 

u

==> small u and v ...

u=5 pixels

Pyramid of image J

u=10 pixels

Pyramid of image I

Optical Flow Results

Optical Flow Results

Lucas-Kanade (1984)

Inherent problems: * Still smoothes motion discontinuities (but unlike Horn & Schunk, does not propagate error across the entire image) * Local singularities (due to the aperture problem) • • Maybe increase the aperture (window) size…?

But no longer a single motion… 

Global parametric motion estimation – next week.

Motion Magnification

Wu, Rubinstein, Shih, Guttag, Durand, Freeman

“Eulerian Video Magnification for Revealing Subtle Changes in the World

”, SIGGRAPH 2012 Source video

: baby.mp4

Result

: baby-iir-r1-0.4-r2-0.05-alpha-10-lambda_c-16-chromAtn-0.1.mp4

Paper + videos can be found on:

http://people.csail.mit.edu/mrub/vidmag

Motion Magnification Could compute optical flow and magnify it But… very complicated (motions are almost invisible) Alternatively:

𝐼 𝑥 ∙ 𝑢 + 𝐼 𝑦 ∙ 𝑣 = 𝐼 𝑡 𝐼 𝑥 ∙ 𝑢

•s

+ 𝐼 𝑦 ∙ 𝑣

•s

= 𝐼 𝑡

•s

But holds only for small u

•s

and v

•s



apply coarse to fine to generate larger motions

𝑰(𝒙, 𝒚)

Motion Magnification

What is 𝐼 𝑡

•s

equivalent to?

𝒕

(time)

 This is equivalent to keeping the same temporal frequencies, but

magnifying the amplitude

(increase frequency coefficient).

 Can decide to do this selectively to specific temporal frequencies (e.g., a range of frequencies of expected heart rates).

Motion Magnification

Wu, Rubinstein, Shih, Guttag, Durand, Freeman

“Eulerian Video Magnification for Revealing Subtle Changes in the World

”, SIGGRAPH 2012

Paper + videos can be found on:

http://people.csail.mit.edu/mrub/vidmag

A simplified version of this work



the next programming exercise

• •

Exercise will be posted within a few days Meanwhile, please read SIGGRAPH’2012 paper