Slide 1

Leow Wee Kheng
CS4243 Computer Vision and Pattern Recognition

Motion Tracking

CS4243

Motion Tracking

1

Changes are everywhere!

CS4243

Motion Tracking

2

Illumination change

CS4243

Motion Tracking

3

Shape change

CS4243

Motion Tracking

4

Object motion

CS4243

Motion Tracking

5

Camera motion

CS4243

Motion Tracking

6

Object & camera motion

CS4243

Motion Tracking

7

Everything changes!

CS4243

Motion Tracking

8

Motion analysis is tough in general!

We focus on object / camera motion.

CS4243

Motion Tracking

9

Change Detection


Detects any change in two video frames.



Straightforward method:
 Compute

difference between corresponding pixels:



: intensity / colour at (x, y) in frame t.
> threshold, has large difference.

 If

CS4243

Motion Tracking

10

Any difference?

No

CS4243

Motion Tracking

11

Any difference?

Yes,
illumination
change

CS4243

Motion Tracking

12

Any difference?

Yes,
position change

CS4243

Motion Tracking

13

Change Detection


Can detect
 Illumination

change
 Position change
 Illumination and position change


But, cannot distinguish between them.



Need to detect and measure position change.

CS4243

Motion Tracking

14

Motion Tracking


Two approaches
 Feature-based
 Intensity

CS4243

gradient-based

Motion Tracking

15

Feature-based Motion Tracking


Look for distinct features that change positions.

 Eagle’s

wing tips change positions.
 Tree tops don’t change positions.
CS4243

Motion Tracking

16

Basic Ideas
1.

Look for distinct features in current frame.

2.

For each feature


Search for matching feature within neighbourhood
in next frame.
 Difference in positions  displacement.
 Velocity = displacement / time difference.

CS4243

Motion Tracking

17

Basic Ideas
displacement

feature area

search area

CS4243

Motion Tracking

18

What feature to use?


Harris corner



Tomasi’s feature



Feature descriptors: SIFT, SURF, GLOH, etc.



Others

CS4243

Motion Tracking

19

Summary


Simple algorithm.



Can be slow if search area is large.



Can constrain search area with prior knowledge.

CS4243

Motion Tracking

20

Gradient-based Motion Tracking


Two basic assumptions
 Intensity

changes smoothly with position.
 Intensity of object doesn’t change over time.


Suppose an object is in motion.
 Change

position (dx, dy) over time dt.
 Then, from 2nd assumption:
 Apply

CS4243

Taylor’s series expansion:

Motion Tracking

21

 Omit

higher order terms and divide by dt

 Denote

 Then,

 u,

v are unknown.
 2 unknowns, 1 equation: can’t solve!

CS4243

Motion Tracking

22

Any difference in motion?

CS4243

Motion Tracking

23

Any difference in motion?

CS4243

Motion Tracking

24

Any difference in motion?

CS4243

Motion Tracking

25

Any difference in motion?

CS4243

Motion Tracking

26

Any difference in motion?

CS4243

Motion Tracking

27

Any difference in motion?

CS4243

Motion Tracking

28

Aperture Problem


Homogeneous region

 Ix

= Iy = It = 0.
 No change in local region.
 Cannot detect motion.

CS4243

Motion Tracking

29

Aperture Problem


Edge

 Ix

and Iy are zero along edge
 Cannot measure motion tangential to edge
 Ix and Iy are non-zero normal to edge
 Can measure motion normal to edge
 So, cannot measure actual motion

CS4243

Motion Tracking

30

Aperture Problem


Corner

 Ix

and Iy are non-zero in two perpendicular directions
 2 unknowns, 2 equations
 Can measure actual motion

CS4243

Motion Tracking

31

Lucas-Kanade Method


Consider two consecutive image frames I and J:
I

J
x+d
d
x

x

moves from x = (x, y)T to x + d.
 d = (u, v) T
 Object

CS4243

Motion Tracking

32



So,



Or



Due to noise, there’s an error at position x:



Sum error over small window W at position x:

Similar to
template matching
CS4243

weight
Motion Tracking

33



If E is small, patterns in I and J match well.



So, find d that minimises E:
E / d = 0, compute d that minimises E.
 First, expand I(x – d) by Taylor’s series expansion:
 Set

 Omit

higher order terms:

 Write

in matrix form:

Intensity gradient
CS4243

Motion Tracking

34

 Now,

error E at position x is:

 Now,

differentiate E with respect to d (exercise):

 Setting

E / d = 0 gives

b

the only
unknown

Z

CS4243

Motion Tracking

35

 So,

2



CS4243

we get

unknowns, 2 equations. Can solve for d = (u, v).

What happen to aperture problem?
Did it disappear?
Motion Tracking

36

Lucas-Kanade + Tomasi


Lucas-Kanade algorithm is often used with
Tomasi’s feature
 Apply

Tomasi’s method to detect good features.
 Apply LK method to compute d for each pixel.
 Accept d only for good features.

CS4243

Motion Tracking

37

Example

Can you spot tracking errors?

CS4243

Motion Tracking

38

Constraints


Math of LK tracker assumes d is small.



In implementation, W is also small.



LK tracker is good only for small displacement.

CS4243

Motion Tracking

39

How to handle large displacement?


What if we scale down images?
 Displacements

CS4243

are smaller!

Motion Tracking

40

Image Pyramid
Usually, smoothen
image with Gaussian
filter before scaling
down.

downsample

scaled image

input image
CS4243

Motion Tracking

41

LK Tracking with Image Pyramid

CS4243

Motion Tracking

42



Construct image pyramids.



Apply LK tracker to low-resolution images.



Propagate results to higher-resolution images.



Apply LK tracker to higher-resolution images.

CS4243

Motion Tracking

43

Example

Tracking results are more accurate.

CS4243

Motion Tracking

44

Summary


Efficient algorithm, no explicit search.



Has aperture problem;
track good features only



LK tracker can’t track large displacement.



Use LK + image pyramid for large displacement.

CS4243

Motion Tracking

45

Software


OpenCV supports LK and LK with pyramid.



[Bir] offers LK with Tomasi’s features & pyramid.

CS4243

Motion Tracking

46

Appendix


Calculation of Ix, Iy, It
 Use

finite difference method
 Forward difference
Ix = I(x  1, y, t)  I(x, y, t)
Iy = I(x, y  1, t)  I(x, y, t)
It = I(x, y, t  1)  I(x, y, t)
 Backward difference
Ix = I(x, y, t)  I(x  1, y, t)
Iy = I(x, y, t)  I(x, y  1, t)
It = I(x, y, t)  I(x, y, t  1)

CS4243

Motion Tracking

47

Further Readings


Lucas-Kanade tracking with pyramid:
[BK08] Chapter 10.



Optical flow: [Sze10] Section 8.4.



Hierarchical motion estimation (with image
pyramid): [Sze10] Section 8.1.1.

CS4243

Motion Tracking

48

References


[Bir] S. Birchfield. KLT: An implementation of the Kanade-LucasTomasi feature tracker. http://vision.stanford.edu/~birch/klt/.



[BK08] Bradski and Kaehler. Learning OpenCV: Computer Vision
with the OpenCV Library. O’Reilly, 2008.



[LK81] B. D. Lucas and T. Kanade. An iterative image registration
technique with an application to stereo vision. In Proceedings of 7th
International Joint Conference on Artificial Intelligence, pages 674–
679, 1981.



[ST94] J. Shi and C. Tomasi. Good features to track. In Proceedings
of IEEE Conference on Computer Vision and Pattern Recognition,
pages 593–600, 1994.



[Sze10] R. Szeliski. Computer Vision: Algorithms and Applications.
Springer, 2010.

CS4243

Motion Tracking

49

References


CS4243

[TK91] C. Tomasi and T. Kanade. Detection and tracking of point
features. Technical Report CMU-CS-91-132, School of Computer
Science, Carnegie Mellon University, 1991.

Motion Tracking

50


Slide 2

Leow Wee Kheng
CS4243 Computer Vision and Pattern Recognition

Motion Tracking

CS4243

Motion Tracking

1

Changes are everywhere!

CS4243

Motion Tracking

2

Illumination change

CS4243

Motion Tracking

3

Shape change

CS4243

Motion Tracking

4

Object motion

CS4243

Motion Tracking

5

Camera motion

CS4243

Motion Tracking

6

Object & camera motion

CS4243

Motion Tracking

7

Everything changes!

CS4243

Motion Tracking

8

Motion analysis is tough in general!

We focus on object / camera motion.

CS4243

Motion Tracking

9

Change Detection


Detects any change in two video frames.



Straightforward method:
 Compute

difference between corresponding pixels:



: intensity / colour at (x, y) in frame t.
> threshold, has large difference.

 If

CS4243

Motion Tracking

10

Any difference?

No

CS4243

Motion Tracking

11

Any difference?

Yes,
illumination
change

CS4243

Motion Tracking

12

Any difference?

Yes,
position change

CS4243

Motion Tracking

13

Change Detection


Can detect
 Illumination

change
 Position change
 Illumination and position change


But, cannot distinguish between them.



Need to detect and measure position change.

CS4243

Motion Tracking

14

Motion Tracking


Two approaches
 Feature-based
 Intensity

CS4243

gradient-based

Motion Tracking

15

Feature-based Motion Tracking


Look for distinct features that change positions.

 Eagle’s

wing tips change positions.
 Tree tops don’t change positions.
CS4243

Motion Tracking

16

Basic Ideas
1.

Look for distinct features in current frame.

2.

For each feature


Search for matching feature within neighbourhood
in next frame.
 Difference in positions  displacement.
 Velocity = displacement / time difference.

CS4243

Motion Tracking

17

Basic Ideas
displacement

feature area

search area

CS4243

Motion Tracking

18

What feature to use?


Harris corner



Tomasi’s feature



Feature descriptors: SIFT, SURF, GLOH, etc.



Others

CS4243

Motion Tracking

19

Summary


Simple algorithm.



Can be slow if search area is large.



Can constrain search area with prior knowledge.

CS4243

Motion Tracking

20

Gradient-based Motion Tracking


Two basic assumptions
 Intensity

changes smoothly with position.
 Intensity of object doesn’t change over time.


Suppose an object is in motion.
 Change

position (dx, dy) over time dt.
 Then, from 2nd assumption:
 Apply

CS4243

Taylor’s series expansion:

Motion Tracking

21

 Omit

higher order terms and divide by dt

 Denote

 Then,

 u,

v are unknown.
 2 unknowns, 1 equation: can’t solve!

CS4243

Motion Tracking

22

Any difference in motion?

CS4243

Motion Tracking

23

Any difference in motion?

CS4243

Motion Tracking

24

Any difference in motion?

CS4243

Motion Tracking

25

Any difference in motion?

CS4243

Motion Tracking

26

Any difference in motion?

CS4243

Motion Tracking

27

Any difference in motion?

CS4243

Motion Tracking

28

Aperture Problem


Homogeneous region

 Ix

= Iy = It = 0.
 No change in local region.
 Cannot detect motion.

CS4243

Motion Tracking

29

Aperture Problem


Edge

 Ix

and Iy are zero along edge
 Cannot measure motion tangential to edge
 Ix and Iy are non-zero normal to edge
 Can measure motion normal to edge
 So, cannot measure actual motion

CS4243

Motion Tracking

30

Aperture Problem


Corner

 Ix

and Iy are non-zero in two perpendicular directions
 2 unknowns, 2 equations
 Can measure actual motion

CS4243

Motion Tracking

31

Lucas-Kanade Method


Consider two consecutive image frames I and J:
I

J
x+d
d
x

x

moves from x = (x, y)T to x + d.
 d = (u, v) T
 Object

CS4243

Motion Tracking

32



So,



Or



Due to noise, there’s an error at position x:



Sum error over small window W at position x:

Similar to
template matching
CS4243

weight
Motion Tracking

33



If E is small, patterns in I and J match well.



So, find d that minimises E:
E / d = 0, compute d that minimises E.
 First, expand I(x – d) by Taylor’s series expansion:
 Set

 Omit

higher order terms:

 Write

in matrix form:

Intensity gradient
CS4243

Motion Tracking

34

 Now,

error E at position x is:

 Now,

differentiate E with respect to d (exercise):

 Setting

E / d = 0 gives

b

the only
unknown

Z

CS4243

Motion Tracking

35

 So,

2



CS4243

we get

unknowns, 2 equations. Can solve for d = (u, v).

What happen to aperture problem?
Did it disappear?
Motion Tracking

36

Lucas-Kanade + Tomasi


Lucas-Kanade algorithm is often used with
Tomasi’s feature
 Apply

Tomasi’s method to detect good features.
 Apply LK method to compute d for each pixel.
 Accept d only for good features.

CS4243

Motion Tracking

37

Example

Can you spot tracking errors?

CS4243

Motion Tracking

38

Constraints


Math of LK tracker assumes d is small.



In implementation, W is also small.



LK tracker is good only for small displacement.

CS4243

Motion Tracking

39

How to handle large displacement?


What if we scale down images?
 Displacements

CS4243

are smaller!

Motion Tracking

40

Image Pyramid
Usually, smoothen
image with Gaussian
filter before scaling
down.

downsample

scaled image

input image
CS4243

Motion Tracking

41

LK Tracking with Image Pyramid

CS4243

Motion Tracking

42



Construct image pyramids.



Apply LK tracker to low-resolution images.



Propagate results to higher-resolution images.



Apply LK tracker to higher-resolution images.

CS4243

Motion Tracking

43

Example

Tracking results are more accurate.

CS4243

Motion Tracking

44

Summary


Efficient algorithm, no explicit search.



Has aperture problem;
track good features only



LK tracker can’t track large displacement.



Use LK + image pyramid for large displacement.

CS4243

Motion Tracking

45

Software


OpenCV supports LK and LK with pyramid.



[Bir] offers LK with Tomasi’s features & pyramid.

CS4243

Motion Tracking

46

Appendix


Calculation of Ix, Iy, It
 Use

finite difference method
 Forward difference
Ix = I(x  1, y, t)  I(x, y, t)
Iy = I(x, y  1, t)  I(x, y, t)
It = I(x, y, t  1)  I(x, y, t)
 Backward difference
Ix = I(x, y, t)  I(x  1, y, t)
Iy = I(x, y, t)  I(x, y  1, t)
It = I(x, y, t)  I(x, y, t  1)

CS4243

Motion Tracking

47

Further Readings


Lucas-Kanade tracking with pyramid:
[BK08] Chapter 10.



Optical flow: [Sze10] Section 8.4.



Hierarchical motion estimation (with image
pyramid): [Sze10] Section 8.1.1.

CS4243

Motion Tracking

48

References


[Bir] S. Birchfield. KLT: An implementation of the Kanade-LucasTomasi feature tracker. http://vision.stanford.edu/~birch/klt/.



[BK08] Bradski and Kaehler. Learning OpenCV: Computer Vision
with the OpenCV Library. O’Reilly, 2008.



[LK81] B. D. Lucas and T. Kanade. An iterative image registration
technique with an application to stereo vision. In Proceedings of 7th
International Joint Conference on Artificial Intelligence, pages 674–
679, 1981.



[ST94] J. Shi and C. Tomasi. Good features to track. In Proceedings
of IEEE Conference on Computer Vision and Pattern Recognition,
pages 593–600, 1994.



[Sze10] R. Szeliski. Computer Vision: Algorithms and Applications.
Springer, 2010.

CS4243

Motion Tracking

49

References


CS4243

[TK91] C. Tomasi and T. Kanade. Detection and tracking of point
features. Technical Report CMU-CS-91-132, School of Computer
Science, Carnegie Mellon University, 1991.

Motion Tracking

50


Slide 3

Leow Wee Kheng
CS4243 Computer Vision and Pattern Recognition

Motion Tracking

CS4243

Motion Tracking

1

Changes are everywhere!

CS4243

Motion Tracking

2

Illumination change

CS4243

Motion Tracking

3

Shape change

CS4243

Motion Tracking

4

Object motion

CS4243

Motion Tracking

5

Camera motion

CS4243

Motion Tracking

6

Object & camera motion

CS4243

Motion Tracking

7

Everything changes!

CS4243

Motion Tracking

8

Motion analysis is tough in general!

We focus on object / camera motion.

CS4243

Motion Tracking

9

Change Detection


Detects any change in two video frames.



Straightforward method:
 Compute

difference between corresponding pixels:



: intensity / colour at (x, y) in frame t.
> threshold, has large difference.

 If

CS4243

Motion Tracking

10

Any difference?

No

CS4243

Motion Tracking

11

Any difference?

Yes,
illumination
change

CS4243

Motion Tracking

12

Any difference?

Yes,
position change

CS4243

Motion Tracking

13

Change Detection


Can detect
 Illumination

change
 Position change
 Illumination and position change


But, cannot distinguish between them.



Need to detect and measure position change.

CS4243

Motion Tracking

14

Motion Tracking


Two approaches
 Feature-based
 Intensity

CS4243

gradient-based

Motion Tracking

15

Feature-based Motion Tracking


Look for distinct features that change positions.

 Eagle’s

wing tips change positions.
 Tree tops don’t change positions.
CS4243

Motion Tracking

16

Basic Ideas
1.

Look for distinct features in current frame.

2.

For each feature


Search for matching feature within neighbourhood
in next frame.
 Difference in positions  displacement.
 Velocity = displacement / time difference.

CS4243

Motion Tracking

17

Basic Ideas
displacement

feature area

search area

CS4243

Motion Tracking

18

What feature to use?


Harris corner



Tomasi’s feature



Feature descriptors: SIFT, SURF, GLOH, etc.



Others

CS4243

Motion Tracking

19

Summary


Simple algorithm.



Can be slow if search area is large.



Can constrain search area with prior knowledge.

CS4243

Motion Tracking

20

Gradient-based Motion Tracking


Two basic assumptions
 Intensity

changes smoothly with position.
 Intensity of object doesn’t change over time.


Suppose an object is in motion.
 Change

position (dx, dy) over time dt.
 Then, from 2nd assumption:
 Apply

CS4243

Taylor’s series expansion:

Motion Tracking

21

 Omit

higher order terms and divide by dt

 Denote

 Then,

 u,

v are unknown.
 2 unknowns, 1 equation: can’t solve!

CS4243

Motion Tracking

22

Any difference in motion?

CS4243

Motion Tracking

23

Any difference in motion?

CS4243

Motion Tracking

24

Any difference in motion?

CS4243

Motion Tracking

25

Any difference in motion?

CS4243

Motion Tracking

26

Any difference in motion?

CS4243

Motion Tracking

27

Any difference in motion?

CS4243

Motion Tracking

28

Aperture Problem


Homogeneous region

 Ix

= Iy = It = 0.
 No change in local region.
 Cannot detect motion.

CS4243

Motion Tracking

29

Aperture Problem


Edge

 Ix

and Iy are zero along edge
 Cannot measure motion tangential to edge
 Ix and Iy are non-zero normal to edge
 Can measure motion normal to edge
 So, cannot measure actual motion

CS4243

Motion Tracking

30

Aperture Problem


Corner

 Ix

and Iy are non-zero in two perpendicular directions
 2 unknowns, 2 equations
 Can measure actual motion

CS4243

Motion Tracking

31

Lucas-Kanade Method


Consider two consecutive image frames I and J:
I

J
x+d
d
x

x

moves from x = (x, y)T to x + d.
 d = (u, v) T
 Object

CS4243

Motion Tracking

32



So,



Or



Due to noise, there’s an error at position x:



Sum error over small window W at position x:

Similar to
template matching
CS4243

weight
Motion Tracking

33



If E is small, patterns in I and J match well.



So, find d that minimises E:
E / d = 0, compute d that minimises E.
 First, expand I(x – d) by Taylor’s series expansion:
 Set

 Omit

higher order terms:

 Write

in matrix form:

Intensity gradient
CS4243

Motion Tracking

34

 Now,

error E at position x is:

 Now,

differentiate E with respect to d (exercise):

 Setting

E / d = 0 gives

b

the only
unknown

Z

CS4243

Motion Tracking

35

 So,

2



CS4243

we get

unknowns, 2 equations. Can solve for d = (u, v).

What happen to aperture problem?
Did it disappear?
Motion Tracking

36

Lucas-Kanade + Tomasi


Lucas-Kanade algorithm is often used with
Tomasi’s feature
 Apply

Tomasi’s method to detect good features.
 Apply LK method to compute d for each pixel.
 Accept d only for good features.

CS4243

Motion Tracking

37

Example

Can you spot tracking errors?

CS4243

Motion Tracking

38

Constraints


Math of LK tracker assumes d is small.



In implementation, W is also small.



LK tracker is good only for small displacement.

CS4243

Motion Tracking

39

How to handle large displacement?


What if we scale down images?
 Displacements

CS4243

are smaller!

Motion Tracking

40

Image Pyramid
Usually, smoothen
image with Gaussian
filter before scaling
down.

downsample

scaled image

input image
CS4243

Motion Tracking

41

LK Tracking with Image Pyramid

CS4243

Motion Tracking

42



Construct image pyramids.



Apply LK tracker to low-resolution images.



Propagate results to higher-resolution images.



Apply LK tracker to higher-resolution images.

CS4243

Motion Tracking

43

Example

Tracking results are more accurate.

CS4243

Motion Tracking

44

Summary


Efficient algorithm, no explicit search.



Has aperture problem;
track good features only



LK tracker can’t track large displacement.



Use LK + image pyramid for large displacement.

CS4243

Motion Tracking

45

Software


OpenCV supports LK and LK with pyramid.



[Bir] offers LK with Tomasi’s features & pyramid.

CS4243

Motion Tracking

46

Appendix


Calculation of Ix, Iy, It
 Use

finite difference method
 Forward difference
Ix = I(x  1, y, t)  I(x, y, t)
Iy = I(x, y  1, t)  I(x, y, t)
It = I(x, y, t  1)  I(x, y, t)
 Backward difference
Ix = I(x, y, t)  I(x  1, y, t)
Iy = I(x, y, t)  I(x, y  1, t)
It = I(x, y, t)  I(x, y, t  1)

CS4243

Motion Tracking

47

Further Readings


Lucas-Kanade tracking with pyramid:
[BK08] Chapter 10.



Optical flow: [Sze10] Section 8.4.



Hierarchical motion estimation (with image
pyramid): [Sze10] Section 8.1.1.

CS4243

Motion Tracking

48

References


[Bir] S. Birchfield. KLT: An implementation of the Kanade-LucasTomasi feature tracker. http://vision.stanford.edu/~birch/klt/.



[BK08] Bradski and Kaehler. Learning OpenCV: Computer Vision
with the OpenCV Library. O’Reilly, 2008.



[LK81] B. D. Lucas and T. Kanade. An iterative image registration
technique with an application to stereo vision. In Proceedings of 7th
International Joint Conference on Artificial Intelligence, pages 674–
679, 1981.



[ST94] J. Shi and C. Tomasi. Good features to track. In Proceedings
of IEEE Conference on Computer Vision and Pattern Recognition,
pages 593–600, 1994.



[Sze10] R. Szeliski. Computer Vision: Algorithms and Applications.
Springer, 2010.

CS4243

Motion Tracking

49

References


CS4243

[TK91] C. Tomasi and T. Kanade. Detection and tracking of point
features. Technical Report CMU-CS-91-132, School of Computer
Science, Carnegie Mellon University, 1991.

Motion Tracking

50


Slide 4

Leow Wee Kheng
CS4243 Computer Vision and Pattern Recognition

Motion Tracking

CS4243

Motion Tracking

1

Changes are everywhere!

CS4243

Motion Tracking

2

Illumination change

CS4243

Motion Tracking

3

Shape change

CS4243

Motion Tracking

4

Object motion

CS4243

Motion Tracking

5

Camera motion

CS4243

Motion Tracking

6

Object & camera motion

CS4243

Motion Tracking

7

Everything changes!

CS4243

Motion Tracking

8

Motion analysis is tough in general!

We focus on object / camera motion.

CS4243

Motion Tracking

9

Change Detection


Detects any change in two video frames.



Straightforward method:
 Compute

difference between corresponding pixels:



: intensity / colour at (x, y) in frame t.
> threshold, has large difference.

 If

CS4243

Motion Tracking

10

Any difference?

No

CS4243

Motion Tracking

11

Any difference?

Yes,
illumination
change

CS4243

Motion Tracking

12

Any difference?

Yes,
position change

CS4243

Motion Tracking

13

Change Detection


Can detect
 Illumination

change
 Position change
 Illumination and position change


But, cannot distinguish between them.



Need to detect and measure position change.

CS4243

Motion Tracking

14

Motion Tracking


Two approaches
 Feature-based
 Intensity

CS4243

gradient-based

Motion Tracking

15

Feature-based Motion Tracking


Look for distinct features that change positions.

 Eagle’s

wing tips change positions.
 Tree tops don’t change positions.
CS4243

Motion Tracking

16

Basic Ideas
1.

Look for distinct features in current frame.

2.

For each feature


Search for matching feature within neighbourhood
in next frame.
 Difference in positions  displacement.
 Velocity = displacement / time difference.

CS4243

Motion Tracking

17

Basic Ideas
displacement

feature area

search area

CS4243

Motion Tracking

18

What feature to use?


Harris corner



Tomasi’s feature



Feature descriptors: SIFT, SURF, GLOH, etc.



Others

CS4243

Motion Tracking

19

Summary


Simple algorithm.



Can be slow if search area is large.



Can constrain search area with prior knowledge.

CS4243

Motion Tracking

20

Gradient-based Motion Tracking


Two basic assumptions
 Intensity

changes smoothly with position.
 Intensity of object doesn’t change over time.


Suppose an object is in motion.
 Change

position (dx, dy) over time dt.
 Then, from 2nd assumption:
 Apply

CS4243

Taylor’s series expansion:

Motion Tracking

21

 Omit

higher order terms and divide by dt

 Denote

 Then,

 u,

v are unknown.
 2 unknowns, 1 equation: can’t solve!

CS4243

Motion Tracking

22

Any difference in motion?

CS4243

Motion Tracking

23

Any difference in motion?

CS4243

Motion Tracking

24

Any difference in motion?

CS4243

Motion Tracking

25

Any difference in motion?

CS4243

Motion Tracking

26

Any difference in motion?

CS4243

Motion Tracking

27

Any difference in motion?

CS4243

Motion Tracking

28

Aperture Problem


Homogeneous region

 Ix

= Iy = It = 0.
 No change in local region.
 Cannot detect motion.

CS4243

Motion Tracking

29

Aperture Problem


Edge

 Ix

and Iy are zero along edge
 Cannot measure motion tangential to edge
 Ix and Iy are non-zero normal to edge
 Can measure motion normal to edge
 So, cannot measure actual motion

CS4243

Motion Tracking

30

Aperture Problem


Corner

 Ix

and Iy are non-zero in two perpendicular directions
 2 unknowns, 2 equations
 Can measure actual motion

CS4243

Motion Tracking

31

Lucas-Kanade Method


Consider two consecutive image frames I and J:
I

J
x+d
d
x

x

moves from x = (x, y)T to x + d.
 d = (u, v) T
 Object

CS4243

Motion Tracking

32



So,



Or



Due to noise, there’s an error at position x:



Sum error over small window W at position x:

Similar to
template matching
CS4243

weight
Motion Tracking

33



If E is small, patterns in I and J match well.



So, find d that minimises E:
E / d = 0, compute d that minimises E.
 First, expand I(x – d) by Taylor’s series expansion:
 Set

 Omit

higher order terms:

 Write

in matrix form:

Intensity gradient
CS4243

Motion Tracking

34

 Now,

error E at position x is:

 Now,

differentiate E with respect to d (exercise):

 Setting

E / d = 0 gives

b

the only
unknown

Z

CS4243

Motion Tracking

35

 So,

2



CS4243

we get

unknowns, 2 equations. Can solve for d = (u, v).

What happen to aperture problem?
Did it disappear?
Motion Tracking

36

Lucas-Kanade + Tomasi


Lucas-Kanade algorithm is often used with
Tomasi’s feature
 Apply

Tomasi’s method to detect good features.
 Apply LK method to compute d for each pixel.
 Accept d only for good features.

CS4243

Motion Tracking

37

Example

Can you spot tracking errors?

CS4243

Motion Tracking

38

Constraints


Math of LK tracker assumes d is small.



In implementation, W is also small.



LK tracker is good only for small displacement.

CS4243

Motion Tracking

39

How to handle large displacement?


What if we scale down images?
 Displacements

CS4243

are smaller!

Motion Tracking

40

Image Pyramid
Usually, smoothen
image with Gaussian
filter before scaling
down.

downsample

scaled image

input image
CS4243

Motion Tracking

41

LK Tracking with Image Pyramid

CS4243

Motion Tracking

42



Construct image pyramids.



Apply LK tracker to low-resolution images.



Propagate results to higher-resolution images.



Apply LK tracker to higher-resolution images.

CS4243

Motion Tracking

43

Example

Tracking results are more accurate.

CS4243

Motion Tracking

44

Summary


Efficient algorithm, no explicit search.



Has aperture problem;
track good features only



LK tracker can’t track large displacement.



Use LK + image pyramid for large displacement.

CS4243

Motion Tracking

45

Software


OpenCV supports LK and LK with pyramid.



[Bir] offers LK with Tomasi’s features & pyramid.

CS4243

Motion Tracking

46

Appendix


Calculation of Ix, Iy, It
 Use

finite difference method
 Forward difference
Ix = I(x  1, y, t)  I(x, y, t)
Iy = I(x, y  1, t)  I(x, y, t)
It = I(x, y, t  1)  I(x, y, t)
 Backward difference
Ix = I(x, y, t)  I(x  1, y, t)
Iy = I(x, y, t)  I(x, y  1, t)
It = I(x, y, t)  I(x, y, t  1)

CS4243

Motion Tracking

47

Further Readings


Lucas-Kanade tracking with pyramid:
[BK08] Chapter 10.



Optical flow: [Sze10] Section 8.4.



Hierarchical motion estimation (with image
pyramid): [Sze10] Section 8.1.1.

CS4243

Motion Tracking

48

References


[Bir] S. Birchfield. KLT: An implementation of the Kanade-LucasTomasi feature tracker. http://vision.stanford.edu/~birch/klt/.



[BK08] Bradski and Kaehler. Learning OpenCV: Computer Vision
with the OpenCV Library. O’Reilly, 2008.



[LK81] B. D. Lucas and T. Kanade. An iterative image registration
technique with an application to stereo vision. In Proceedings of 7th
International Joint Conference on Artificial Intelligence, pages 674–
679, 1981.



[ST94] J. Shi and C. Tomasi. Good features to track. In Proceedings
of IEEE Conference on Computer Vision and Pattern Recognition,
pages 593–600, 1994.



[Sze10] R. Szeliski. Computer Vision: Algorithms and Applications.
Springer, 2010.

CS4243

Motion Tracking

49

References


CS4243

[TK91] C. Tomasi and T. Kanade. Detection and tracking of point
features. Technical Report CMU-CS-91-132, School of Computer
Science, Carnegie Mellon University, 1991.

Motion Tracking

50


Slide 5

Leow Wee Kheng
CS4243 Computer Vision and Pattern Recognition

Motion Tracking

CS4243

Motion Tracking

1

Changes are everywhere!

CS4243

Motion Tracking

2

Illumination change

CS4243

Motion Tracking

3

Shape change

CS4243

Motion Tracking

4

Object motion

CS4243

Motion Tracking

5

Camera motion

CS4243

Motion Tracking

6

Object & camera motion

CS4243

Motion Tracking

7

Everything changes!

CS4243

Motion Tracking

8

Motion analysis is tough in general!

We focus on object / camera motion.

CS4243

Motion Tracking

9

Change Detection


Detects any change in two video frames.



Straightforward method:
 Compute

difference between corresponding pixels:



: intensity / colour at (x, y) in frame t.
> threshold, has large difference.

 If

CS4243

Motion Tracking

10

Any difference?

No

CS4243

Motion Tracking

11

Any difference?

Yes,
illumination
change

CS4243

Motion Tracking

12

Any difference?

Yes,
position change

CS4243

Motion Tracking

13

Change Detection


Can detect
 Illumination

change
 Position change
 Illumination and position change


But, cannot distinguish between them.



Need to detect and measure position change.

CS4243

Motion Tracking

14

Motion Tracking


Two approaches
 Feature-based
 Intensity

CS4243

gradient-based

Motion Tracking

15

Feature-based Motion Tracking


Look for distinct features that change positions.

 Eagle’s

wing tips change positions.
 Tree tops don’t change positions.
CS4243

Motion Tracking

16

Basic Ideas
1.

Look for distinct features in current frame.

2.

For each feature


Search for matching feature within neighbourhood
in next frame.
 Difference in positions  displacement.
 Velocity = displacement / time difference.

CS4243

Motion Tracking

17

Basic Ideas
displacement

feature area

search area

CS4243

Motion Tracking

18

What feature to use?


Harris corner



Tomasi’s feature



Feature descriptors: SIFT, SURF, GLOH, etc.



Others

CS4243

Motion Tracking

19

Summary


Simple algorithm.



Can be slow if search area is large.



Can constrain search area with prior knowledge.

CS4243

Motion Tracking

20

Gradient-based Motion Tracking


Two basic assumptions
 Intensity

changes smoothly with position.
 Intensity of object doesn’t change over time.


Suppose an object is in motion.
 Change

position (dx, dy) over time dt.
 Then, from 2nd assumption:
 Apply

CS4243

Taylor’s series expansion:

Motion Tracking

21

 Omit

higher order terms and divide by dt

 Denote

 Then,

 u,

v are unknown.
 2 unknowns, 1 equation: can’t solve!

CS4243

Motion Tracking

22

Any difference in motion?

CS4243

Motion Tracking

23

Any difference in motion?

CS4243

Motion Tracking

24

Any difference in motion?

CS4243

Motion Tracking

25

Any difference in motion?

CS4243

Motion Tracking

26

Any difference in motion?

CS4243

Motion Tracking

27

Any difference in motion?

CS4243

Motion Tracking

28

Aperture Problem


Homogeneous region

 Ix

= Iy = It = 0.
 No change in local region.
 Cannot detect motion.

CS4243

Motion Tracking

29

Aperture Problem


Edge

 Ix

and Iy are zero along edge
 Cannot measure motion tangential to edge
 Ix and Iy are non-zero normal to edge
 Can measure motion normal to edge
 So, cannot measure actual motion

CS4243

Motion Tracking

30

Aperture Problem


Corner

 Ix

and Iy are non-zero in two perpendicular directions
 2 unknowns, 2 equations
 Can measure actual motion

CS4243

Motion Tracking

31

Lucas-Kanade Method


Consider two consecutive image frames I and J:
I

J
x+d
d
x

x

moves from x = (x, y)T to x + d.
 d = (u, v) T
 Object

CS4243

Motion Tracking

32



So,



Or



Due to noise, there’s an error at position x:



Sum error over small window W at position x:

Similar to
template matching
CS4243

weight
Motion Tracking

33



If E is small, patterns in I and J match well.



So, find d that minimises E:
E / d = 0, compute d that minimises E.
 First, expand I(x – d) by Taylor’s series expansion:
 Set

 Omit

higher order terms:

 Write

in matrix form:

Intensity gradient
CS4243

Motion Tracking

34

 Now,

error E at position x is:

 Now,

differentiate E with respect to d (exercise):

 Setting

E / d = 0 gives

b

the only
unknown

Z

CS4243

Motion Tracking

35

 So,

2



CS4243

we get

unknowns, 2 equations. Can solve for d = (u, v).

What happen to aperture problem?
Did it disappear?
Motion Tracking

36

Lucas-Kanade + Tomasi


Lucas-Kanade algorithm is often used with
Tomasi’s feature
 Apply

Tomasi’s method to detect good features.
 Apply LK method to compute d for each pixel.
 Accept d only for good features.

CS4243

Motion Tracking

37

Example

Can you spot tracking errors?

CS4243

Motion Tracking

38

Constraints


Math of LK tracker assumes d is small.



In implementation, W is also small.



LK tracker is good only for small displacement.

CS4243

Motion Tracking

39

How to handle large displacement?


What if we scale down images?
 Displacements

CS4243

are smaller!

Motion Tracking

40

Image Pyramid
Usually, smoothen
image with Gaussian
filter before scaling
down.

downsample

scaled image

input image
CS4243

Motion Tracking

41

LK Tracking with Image Pyramid

CS4243

Motion Tracking

42



Construct image pyramids.



Apply LK tracker to low-resolution images.



Propagate results to higher-resolution images.



Apply LK tracker to higher-resolution images.

CS4243

Motion Tracking

43

Example

Tracking results are more accurate.

CS4243

Motion Tracking

44

Summary


Efficient algorithm, no explicit search.



Has aperture problem;
track good features only



LK tracker can’t track large displacement.



Use LK + image pyramid for large displacement.

CS4243

Motion Tracking

45

Software


OpenCV supports LK and LK with pyramid.



[Bir] offers LK with Tomasi’s features & pyramid.

CS4243

Motion Tracking

46

Appendix


Calculation of Ix, Iy, It
 Use

finite difference method
 Forward difference
Ix = I(x  1, y, t)  I(x, y, t)
Iy = I(x, y  1, t)  I(x, y, t)
It = I(x, y, t  1)  I(x, y, t)
 Backward difference
Ix = I(x, y, t)  I(x  1, y, t)
Iy = I(x, y, t)  I(x, y  1, t)
It = I(x, y, t)  I(x, y, t  1)

CS4243

Motion Tracking

47

Further Readings


Lucas-Kanade tracking with pyramid:
[BK08] Chapter 10.



Optical flow: [Sze10] Section 8.4.



Hierarchical motion estimation (with image
pyramid): [Sze10] Section 8.1.1.

CS4243

Motion Tracking

48

References


[Bir] S. Birchfield. KLT: An implementation of the Kanade-LucasTomasi feature tracker. http://vision.stanford.edu/~birch/klt/.



[BK08] Bradski and Kaehler. Learning OpenCV: Computer Vision
with the OpenCV Library. O’Reilly, 2008.



[LK81] B. D. Lucas and T. Kanade. An iterative image registration
technique with an application to stereo vision. In Proceedings of 7th
International Joint Conference on Artificial Intelligence, pages 674–
679, 1981.



[ST94] J. Shi and C. Tomasi. Good features to track. In Proceedings
of IEEE Conference on Computer Vision and Pattern Recognition,
pages 593–600, 1994.



[Sze10] R. Szeliski. Computer Vision: Algorithms and Applications.
Springer, 2010.

CS4243

Motion Tracking

49

References


CS4243

[TK91] C. Tomasi and T. Kanade. Detection and tracking of point
features. Technical Report CMU-CS-91-132, School of Computer
Science, Carnegie Mellon University, 1991.

Motion Tracking

50


Slide 6

Leow Wee Kheng
CS4243 Computer Vision and Pattern Recognition

Motion Tracking

CS4243

Motion Tracking

1

Changes are everywhere!

CS4243

Motion Tracking

2

Illumination change

CS4243

Motion Tracking

3

Shape change

CS4243

Motion Tracking

4

Object motion

CS4243

Motion Tracking

5

Camera motion

CS4243

Motion Tracking

6

Object & camera motion

CS4243

Motion Tracking

7

Everything changes!

CS4243

Motion Tracking

8

Motion analysis is tough in general!

We focus on object / camera motion.

CS4243

Motion Tracking

9

Change Detection


Detects any change in two video frames.



Straightforward method:
 Compute

difference between corresponding pixels:



: intensity / colour at (x, y) in frame t.
> threshold, has large difference.

 If

CS4243

Motion Tracking

10

Any difference?

No

CS4243

Motion Tracking

11

Any difference?

Yes,
illumination
change

CS4243

Motion Tracking

12

Any difference?

Yes,
position change

CS4243

Motion Tracking

13

Change Detection


Can detect
 Illumination

change
 Position change
 Illumination and position change


But, cannot distinguish between them.



Need to detect and measure position change.

CS4243

Motion Tracking

14

Motion Tracking


Two approaches
 Feature-based
 Intensity

CS4243

gradient-based

Motion Tracking

15

Feature-based Motion Tracking


Look for distinct features that change positions.

 Eagle’s

wing tips change positions.
 Tree tops don’t change positions.
CS4243

Motion Tracking

16

Basic Ideas
1.

Look for distinct features in current frame.

2.

For each feature


Search for matching feature within neighbourhood
in next frame.
 Difference in positions  displacement.
 Velocity = displacement / time difference.

CS4243

Motion Tracking

17

Basic Ideas
displacement

feature area

search area

CS4243

Motion Tracking

18

What feature to use?


Harris corner



Tomasi’s feature



Feature descriptors: SIFT, SURF, GLOH, etc.



Others

CS4243

Motion Tracking

19

Summary


Simple algorithm.



Can be slow if search area is large.



Can constrain search area with prior knowledge.

CS4243

Motion Tracking

20

Gradient-based Motion Tracking


Two basic assumptions
 Intensity

changes smoothly with position.
 Intensity of object doesn’t change over time.


Suppose an object is in motion.
 Change

position (dx, dy) over time dt.
 Then, from 2nd assumption:
 Apply

CS4243

Taylor’s series expansion:

Motion Tracking

21

 Omit

higher order terms and divide by dt

 Denote

 Then,

 u,

v are unknown.
 2 unknowns, 1 equation: can’t solve!

CS4243

Motion Tracking

22

Any difference in motion?

CS4243

Motion Tracking

23

Any difference in motion?

CS4243

Motion Tracking

24

Any difference in motion?

CS4243

Motion Tracking

25

Any difference in motion?

CS4243

Motion Tracking

26

Any difference in motion?

CS4243

Motion Tracking

27

Any difference in motion?

CS4243

Motion Tracking

28

Aperture Problem


Homogeneous region

 Ix

= Iy = It = 0.
 No change in local region.
 Cannot detect motion.

CS4243

Motion Tracking

29

Aperture Problem


Edge

 Ix

and Iy are zero along edge
 Cannot measure motion tangential to edge
 Ix and Iy are non-zero normal to edge
 Can measure motion normal to edge
 So, cannot measure actual motion

CS4243

Motion Tracking

30

Aperture Problem


Corner

 Ix

and Iy are non-zero in two perpendicular directions
 2 unknowns, 2 equations
 Can measure actual motion

CS4243

Motion Tracking

31

Lucas-Kanade Method


Consider two consecutive image frames I and J:
I

J
x+d
d
x

x

moves from x = (x, y)T to x + d.
 d = (u, v) T
 Object

CS4243

Motion Tracking

32



So,



Or



Due to noise, there’s an error at position x:



Sum error over small window W at position x:

Similar to
template matching
CS4243

weight
Motion Tracking

33



If E is small, patterns in I and J match well.



So, find d that minimises E:
E / d = 0, compute d that minimises E.
 First, expand I(x – d) by Taylor’s series expansion:
 Set

 Omit

higher order terms:

 Write

in matrix form:

Intensity gradient
CS4243

Motion Tracking

34

 Now,

error E at position x is:

 Now,

differentiate E with respect to d (exercise):

 Setting

E / d = 0 gives

b

the only
unknown

Z

CS4243

Motion Tracking

35

 So,

2



CS4243

we get

unknowns, 2 equations. Can solve for d = (u, v).

What happen to aperture problem?
Did it disappear?
Motion Tracking

36

Lucas-Kanade + Tomasi


Lucas-Kanade algorithm is often used with
Tomasi’s feature
 Apply

Tomasi’s method to detect good features.
 Apply LK method to compute d for each pixel.
 Accept d only for good features.

CS4243

Motion Tracking

37

Example

Can you spot tracking errors?

CS4243

Motion Tracking

38

Constraints


Math of LK tracker assumes d is small.



In implementation, W is also small.



LK tracker is good only for small displacement.

CS4243

Motion Tracking

39

How to handle large displacement?


What if we scale down images?
 Displacements

CS4243

are smaller!

Motion Tracking

40

Image Pyramid
Usually, smoothen
image with Gaussian
filter before scaling
down.

downsample

scaled image

input image
CS4243

Motion Tracking

41

LK Tracking with Image Pyramid

CS4243

Motion Tracking

42



Construct image pyramids.



Apply LK tracker to low-resolution images.



Propagate results to higher-resolution images.



Apply LK tracker to higher-resolution images.

CS4243

Motion Tracking

43

Example

Tracking results are more accurate.

CS4243

Motion Tracking

44

Summary


Efficient algorithm, no explicit search.



Has aperture problem;
track good features only



LK tracker can’t track large displacement.



Use LK + image pyramid for large displacement.

CS4243

Motion Tracking

45

Software


OpenCV supports LK and LK with pyramid.



[Bir] offers LK with Tomasi’s features & pyramid.

CS4243

Motion Tracking

46

Appendix


Calculation of Ix, Iy, It
 Use

finite difference method
 Forward difference
Ix = I(x  1, y, t)  I(x, y, t)
Iy = I(x, y  1, t)  I(x, y, t)
It = I(x, y, t  1)  I(x, y, t)
 Backward difference
Ix = I(x, y, t)  I(x  1, y, t)
Iy = I(x, y, t)  I(x, y  1, t)
It = I(x, y, t)  I(x, y, t  1)

CS4243

Motion Tracking

47

Further Readings


Lucas-Kanade tracking with pyramid:
[BK08] Chapter 10.



Optical flow: [Sze10] Section 8.4.



Hierarchical motion estimation (with image
pyramid): [Sze10] Section 8.1.1.

CS4243

Motion Tracking

48

References


[Bir] S. Birchfield. KLT: An implementation of the Kanade-LucasTomasi feature tracker. http://vision.stanford.edu/~birch/klt/.



[BK08] Bradski and Kaehler. Learning OpenCV: Computer Vision
with the OpenCV Library. O’Reilly, 2008.



[LK81] B. D. Lucas and T. Kanade. An iterative image registration
technique with an application to stereo vision. In Proceedings of 7th
International Joint Conference on Artificial Intelligence, pages 674–
679, 1981.



[ST94] J. Shi and C. Tomasi. Good features to track. In Proceedings
of IEEE Conference on Computer Vision and Pattern Recognition,
pages 593–600, 1994.



[Sze10] R. Szeliski. Computer Vision: Algorithms and Applications.
Springer, 2010.

CS4243

Motion Tracking

49

References


CS4243

[TK91] C. Tomasi and T. Kanade. Detection and tracking of point
features. Technical Report CMU-CS-91-132, School of Computer
Science, Carnegie Mellon University, 1991.

Motion Tracking

50


Slide 7

Leow Wee Kheng
CS4243 Computer Vision and Pattern Recognition

Motion Tracking

CS4243

Motion Tracking

1

Changes are everywhere!

CS4243

Motion Tracking

2

Illumination change

CS4243

Motion Tracking

3

Shape change

CS4243

Motion Tracking

4

Object motion

CS4243

Motion Tracking

5

Camera motion

CS4243

Motion Tracking

6

Object & camera motion

CS4243

Motion Tracking

7

Everything changes!

CS4243

Motion Tracking

8

Motion analysis is tough in general!

We focus on object / camera motion.

CS4243

Motion Tracking

9

Change Detection


Detects any change in two video frames.



Straightforward method:
 Compute

difference between corresponding pixels:



: intensity / colour at (x, y) in frame t.
> threshold, has large difference.

 If

CS4243

Motion Tracking

10

Any difference?

No

CS4243

Motion Tracking

11

Any difference?

Yes,
illumination
change

CS4243

Motion Tracking

12

Any difference?

Yes,
position change

CS4243

Motion Tracking

13

Change Detection


Can detect
 Illumination

change
 Position change
 Illumination and position change


But, cannot distinguish between them.



Need to detect and measure position change.

CS4243

Motion Tracking

14

Motion Tracking


Two approaches
 Feature-based
 Intensity

CS4243

gradient-based

Motion Tracking

15

Feature-based Motion Tracking


Look for distinct features that change positions.

 Eagle’s

wing tips change positions.
 Tree tops don’t change positions.
CS4243

Motion Tracking

16

Basic Ideas
1.

Look for distinct features in current frame.

2.

For each feature


Search for matching feature within neighbourhood
in next frame.
 Difference in positions  displacement.
 Velocity = displacement / time difference.

CS4243

Motion Tracking

17

Basic Ideas
displacement

feature area

search area

CS4243

Motion Tracking

18

What feature to use?


Harris corner



Tomasi’s feature



Feature descriptors: SIFT, SURF, GLOH, etc.



Others

CS4243

Motion Tracking

19

Summary


Simple algorithm.



Can be slow if search area is large.



Can constrain search area with prior knowledge.

CS4243

Motion Tracking

20

Gradient-based Motion Tracking


Two basic assumptions
 Intensity

changes smoothly with position.
 Intensity of object doesn’t change over time.


Suppose an object is in motion.
 Change

position (dx, dy) over time dt.
 Then, from 2nd assumption:
 Apply

CS4243

Taylor’s series expansion:

Motion Tracking

21

 Omit

higher order terms and divide by dt

 Denote

 Then,

 u,

v are unknown.
 2 unknowns, 1 equation: can’t solve!

CS4243

Motion Tracking

22

Any difference in motion?

CS4243

Motion Tracking

23

Any difference in motion?

CS4243

Motion Tracking

24

Any difference in motion?

CS4243

Motion Tracking

25

Any difference in motion?

CS4243

Motion Tracking

26

Any difference in motion?

CS4243

Motion Tracking

27

Any difference in motion?

CS4243

Motion Tracking

28

Aperture Problem


Homogeneous region

 Ix

= Iy = It = 0.
 No change in local region.
 Cannot detect motion.

CS4243

Motion Tracking

29

Aperture Problem


Edge

 Ix

and Iy are zero along edge
 Cannot measure motion tangential to edge
 Ix and Iy are non-zero normal to edge
 Can measure motion normal to edge
 So, cannot measure actual motion

CS4243

Motion Tracking

30

Aperture Problem


Corner

 Ix

and Iy are non-zero in two perpendicular directions
 2 unknowns, 2 equations
 Can measure actual motion

CS4243

Motion Tracking

31

Lucas-Kanade Method


Consider two consecutive image frames I and J:
I

J
x+d
d
x

x

moves from x = (x, y)T to x + d.
 d = (u, v) T
 Object

CS4243

Motion Tracking

32



So,



Or



Due to noise, there’s an error at position x:



Sum error over small window W at position x:

Similar to
template matching
CS4243

weight
Motion Tracking

33



If E is small, patterns in I and J match well.



So, find d that minimises E:
E / d = 0, compute d that minimises E.
 First, expand I(x – d) by Taylor’s series expansion:
 Set

 Omit

higher order terms:

 Write

in matrix form:

Intensity gradient
CS4243

Motion Tracking

34

 Now,

error E at position x is:

 Now,

differentiate E with respect to d (exercise):

 Setting

E / d = 0 gives

b

the only
unknown

Z

CS4243

Motion Tracking

35

 So,

2



CS4243

we get

unknowns, 2 equations. Can solve for d = (u, v).

What happen to aperture problem?
Did it disappear?
Motion Tracking

36

Lucas-Kanade + Tomasi


Lucas-Kanade algorithm is often used with
Tomasi’s feature
 Apply

Tomasi’s method to detect good features.
 Apply LK method to compute d for each pixel.
 Accept d only for good features.

CS4243

Motion Tracking

37

Example

Can you spot tracking errors?

CS4243

Motion Tracking

38

Constraints


Math of LK tracker assumes d is small.



In implementation, W is also small.



LK tracker is good only for small displacement.

CS4243

Motion Tracking

39

How to handle large displacement?


What if we scale down images?
 Displacements

CS4243

are smaller!

Motion Tracking

40

Image Pyramid
Usually, smoothen
image with Gaussian
filter before scaling
down.

downsample

scaled image

input image
CS4243

Motion Tracking

41

LK Tracking with Image Pyramid

CS4243

Motion Tracking

42



Construct image pyramids.



Apply LK tracker to low-resolution images.



Propagate results to higher-resolution images.



Apply LK tracker to higher-resolution images.

CS4243

Motion Tracking

43

Example

Tracking results are more accurate.

CS4243

Motion Tracking

44

Summary


Efficient algorithm, no explicit search.



Has aperture problem;
track good features only



LK tracker can’t track large displacement.



Use LK + image pyramid for large displacement.

CS4243

Motion Tracking

45

Software


OpenCV supports LK and LK with pyramid.



[Bir] offers LK with Tomasi’s features & pyramid.

CS4243

Motion Tracking

46

Appendix


Calculation of Ix, Iy, It
 Use

finite difference method
 Forward difference
Ix = I(x  1, y, t)  I(x, y, t)
Iy = I(x, y  1, t)  I(x, y, t)
It = I(x, y, t  1)  I(x, y, t)
 Backward difference
Ix = I(x, y, t)  I(x  1, y, t)
Iy = I(x, y, t)  I(x, y  1, t)
It = I(x, y, t)  I(x, y, t  1)

CS4243

Motion Tracking

47

Further Readings


Lucas-Kanade tracking with pyramid:
[BK08] Chapter 10.



Optical flow: [Sze10] Section 8.4.



Hierarchical motion estimation (with image
pyramid): [Sze10] Section 8.1.1.

CS4243

Motion Tracking

48

References


[Bir] S. Birchfield. KLT: An implementation of the Kanade-LucasTomasi feature tracker. http://vision.stanford.edu/~birch/klt/.



[BK08] Bradski and Kaehler. Learning OpenCV: Computer Vision
with the OpenCV Library. O’Reilly, 2008.



[LK81] B. D. Lucas and T. Kanade. An iterative image registration
technique with an application to stereo vision. In Proceedings of 7th
International Joint Conference on Artificial Intelligence, pages 674–
679, 1981.



[ST94] J. Shi and C. Tomasi. Good features to track. In Proceedings
of IEEE Conference on Computer Vision and Pattern Recognition,
pages 593–600, 1994.



[Sze10] R. Szeliski. Computer Vision: Algorithms and Applications.
Springer, 2010.

CS4243

Motion Tracking

49

References


CS4243

[TK91] C. Tomasi and T. Kanade. Detection and tracking of point
features. Technical Report CMU-CS-91-132, School of Computer
Science, Carnegie Mellon University, 1991.

Motion Tracking

50


Slide 8

Leow Wee Kheng
CS4243 Computer Vision and Pattern Recognition

Motion Tracking

CS4243

Motion Tracking

1

Changes are everywhere!

CS4243

Motion Tracking

2

Illumination change

CS4243

Motion Tracking

3

Shape change

CS4243

Motion Tracking

4

Object motion

CS4243

Motion Tracking

5

Camera motion

CS4243

Motion Tracking

6

Object & camera motion

CS4243

Motion Tracking

7

Everything changes!

CS4243

Motion Tracking

8

Motion analysis is tough in general!

We focus on object / camera motion.

CS4243

Motion Tracking

9

Change Detection


Detects any change in two video frames.



Straightforward method:
 Compute

difference between corresponding pixels:



: intensity / colour at (x, y) in frame t.
> threshold, has large difference.

 If

CS4243

Motion Tracking

10

Any difference?

No

CS4243

Motion Tracking

11

Any difference?

Yes,
illumination
change

CS4243

Motion Tracking

12

Any difference?

Yes,
position change

CS4243

Motion Tracking

13

Change Detection


Can detect
 Illumination

change
 Position change
 Illumination and position change


But, cannot distinguish between them.



Need to detect and measure position change.

CS4243

Motion Tracking

14

Motion Tracking


Two approaches
 Feature-based
 Intensity

CS4243

gradient-based

Motion Tracking

15

Feature-based Motion Tracking


Look for distinct features that change positions.

 Eagle’s

wing tips change positions.
 Tree tops don’t change positions.
CS4243

Motion Tracking

16

Basic Ideas
1.

Look for distinct features in current frame.

2.

For each feature


Search for matching feature within neighbourhood
in next frame.
 Difference in positions  displacement.
 Velocity = displacement / time difference.

CS4243

Motion Tracking

17

Basic Ideas
displacement

feature area

search area

CS4243

Motion Tracking

18

What feature to use?


Harris corner



Tomasi’s feature



Feature descriptors: SIFT, SURF, GLOH, etc.



Others

CS4243

Motion Tracking

19

Summary


Simple algorithm.



Can be slow if search area is large.



Can constrain search area with prior knowledge.

CS4243

Motion Tracking

20

Gradient-based Motion Tracking


Two basic assumptions
 Intensity

changes smoothly with position.
 Intensity of object doesn’t change over time.


Suppose an object is in motion.
 Change

position (dx, dy) over time dt.
 Then, from 2nd assumption:
 Apply

CS4243

Taylor’s series expansion:

Motion Tracking

21

 Omit

higher order terms and divide by dt

 Denote

 Then,

 u,

v are unknown.
 2 unknowns, 1 equation: can’t solve!

CS4243

Motion Tracking

22

Any difference in motion?

CS4243

Motion Tracking

23

Any difference in motion?

CS4243

Motion Tracking

24

Any difference in motion?

CS4243

Motion Tracking

25

Any difference in motion?

CS4243

Motion Tracking

26

Any difference in motion?

CS4243

Motion Tracking

27

Any difference in motion?

CS4243

Motion Tracking

28

Aperture Problem


Homogeneous region

 Ix

= Iy = It = 0.
 No change in local region.
 Cannot detect motion.

CS4243

Motion Tracking

29

Aperture Problem


Edge

 Ix

and Iy are zero along edge
 Cannot measure motion tangential to edge
 Ix and Iy are non-zero normal to edge
 Can measure motion normal to edge
 So, cannot measure actual motion

CS4243

Motion Tracking

30

Aperture Problem


Corner

 Ix

and Iy are non-zero in two perpendicular directions
 2 unknowns, 2 equations
 Can measure actual motion

CS4243

Motion Tracking

31

Lucas-Kanade Method


Consider two consecutive image frames I and J:
I

J
x+d
d
x

x

moves from x = (x, y)T to x + d.
 d = (u, v) T
 Object

CS4243

Motion Tracking

32



So,



Or



Due to noise, there’s an error at position x:



Sum error over small window W at position x:

Similar to
template matching
CS4243

weight
Motion Tracking

33



If E is small, patterns in I and J match well.



So, find d that minimises E:
E / d = 0, compute d that minimises E.
 First, expand I(x – d) by Taylor’s series expansion:
 Set

 Omit

higher order terms:

 Write

in matrix form:

Intensity gradient
CS4243

Motion Tracking

34

 Now,

error E at position x is:

 Now,

differentiate E with respect to d (exercise):

 Setting

E / d = 0 gives

b

the only
unknown

Z

CS4243

Motion Tracking

35

 So,

2



CS4243

we get

unknowns, 2 equations. Can solve for d = (u, v).

What happen to aperture problem?
Did it disappear?
Motion Tracking

36

Lucas-Kanade + Tomasi


Lucas-Kanade algorithm is often used with
Tomasi’s feature
 Apply

Tomasi’s method to detect good features.
 Apply LK method to compute d for each pixel.
 Accept d only for good features.

CS4243

Motion Tracking

37

Example

Can you spot tracking errors?

CS4243

Motion Tracking

38

Constraints


Math of LK tracker assumes d is small.



In implementation, W is also small.



LK tracker is good only for small displacement.

CS4243

Motion Tracking

39

How to handle large displacement?


What if we scale down images?
 Displacements

CS4243

are smaller!

Motion Tracking

40

Image Pyramid
Usually, smoothen
image with Gaussian
filter before scaling
down.

downsample

scaled image

input image
CS4243

Motion Tracking

41

LK Tracking with Image Pyramid

CS4243

Motion Tracking

42



Construct image pyramids.



Apply LK tracker to low-resolution images.



Propagate results to higher-resolution images.



Apply LK tracker to higher-resolution images.

CS4243

Motion Tracking

43

Example

Tracking results are more accurate.

CS4243

Motion Tracking

44

Summary


Efficient algorithm, no explicit search.



Has aperture problem;
track good features only



LK tracker can’t track large displacement.



Use LK + image pyramid for large displacement.

CS4243

Motion Tracking

45

Software


OpenCV supports LK and LK with pyramid.



[Bir] offers LK with Tomasi’s features & pyramid.

CS4243

Motion Tracking

46

Appendix


Calculation of Ix, Iy, It
 Use

finite difference method
 Forward difference
Ix = I(x  1, y, t)  I(x, y, t)
Iy = I(x, y  1, t)  I(x, y, t)
It = I(x, y, t  1)  I(x, y, t)
 Backward difference
Ix = I(x, y, t)  I(x  1, y, t)
Iy = I(x, y, t)  I(x, y  1, t)
It = I(x, y, t)  I(x, y, t  1)

CS4243

Motion Tracking

47

Further Readings


Lucas-Kanade tracking with pyramid:
[BK08] Chapter 10.



Optical flow: [Sze10] Section 8.4.



Hierarchical motion estimation (with image
pyramid): [Sze10] Section 8.1.1.

CS4243

Motion Tracking

48

References


[Bir] S. Birchfield. KLT: An implementation of the Kanade-LucasTomasi feature tracker. http://vision.stanford.edu/~birch/klt/.



[BK08] Bradski and Kaehler. Learning OpenCV: Computer Vision
with the OpenCV Library. O’Reilly, 2008.



[LK81] B. D. Lucas and T. Kanade. An iterative image registration
technique with an application to stereo vision. In Proceedings of 7th
International Joint Conference on Artificial Intelligence, pages 674–
679, 1981.



[ST94] J. Shi and C. Tomasi. Good features to track. In Proceedings
of IEEE Conference on Computer Vision and Pattern Recognition,
pages 593–600, 1994.



[Sze10] R. Szeliski. Computer Vision: Algorithms and Applications.
Springer, 2010.

CS4243

Motion Tracking

49

References


CS4243

[TK91] C. Tomasi and T. Kanade. Detection and tracking of point
features. Technical Report CMU-CS-91-132, School of Computer
Science, Carnegie Mellon University, 1991.

Motion Tracking

50


Slide 9

Leow Wee Kheng
CS4243 Computer Vision and Pattern Recognition

Motion Tracking

CS4243

Motion Tracking

1

Changes are everywhere!

CS4243

Motion Tracking

2

Illumination change

CS4243

Motion Tracking

3

Shape change

CS4243

Motion Tracking

4

Object motion

CS4243

Motion Tracking

5

Camera motion

CS4243

Motion Tracking

6

Object & camera motion

CS4243

Motion Tracking

7

Everything changes!

CS4243

Motion Tracking

8

Motion analysis is tough in general!

We focus on object / camera motion.

CS4243

Motion Tracking

9

Change Detection


Detects any change in two video frames.



Straightforward method:
 Compute

difference between corresponding pixels:



: intensity / colour at (x, y) in frame t.
> threshold, has large difference.

 If

CS4243

Motion Tracking

10

Any difference?

No

CS4243

Motion Tracking

11

Any difference?

Yes,
illumination
change

CS4243

Motion Tracking

12

Any difference?

Yes,
position change

CS4243

Motion Tracking

13

Change Detection


Can detect
 Illumination

change
 Position change
 Illumination and position change


But, cannot distinguish between them.



Need to detect and measure position change.

CS4243

Motion Tracking

14

Motion Tracking


Two approaches
 Feature-based
 Intensity

CS4243

gradient-based

Motion Tracking

15

Feature-based Motion Tracking


Look for distinct features that change positions.

 Eagle’s

wing tips change positions.
 Tree tops don’t change positions.
CS4243

Motion Tracking

16

Basic Ideas
1.

Look for distinct features in current frame.

2.

For each feature


Search for matching feature within neighbourhood
in next frame.
 Difference in positions  displacement.
 Velocity = displacement / time difference.

CS4243

Motion Tracking

17

Basic Ideas
displacement

feature area

search area

CS4243

Motion Tracking

18

What feature to use?


Harris corner



Tomasi’s feature



Feature descriptors: SIFT, SURF, GLOH, etc.



Others

CS4243

Motion Tracking

19

Summary


Simple algorithm.



Can be slow if search area is large.



Can constrain search area with prior knowledge.

CS4243

Motion Tracking

20

Gradient-based Motion Tracking


Two basic assumptions
 Intensity

changes smoothly with position.
 Intensity of object doesn’t change over time.


Suppose an object is in motion.
 Change

position (dx, dy) over time dt.
 Then, from 2nd assumption:
 Apply

CS4243

Taylor’s series expansion:

Motion Tracking

21

 Omit

higher order terms and divide by dt

 Denote

 Then,

 u,

v are unknown.
 2 unknowns, 1 equation: can’t solve!

CS4243

Motion Tracking

22

Any difference in motion?

CS4243

Motion Tracking

23

Any difference in motion?

CS4243

Motion Tracking

24

Any difference in motion?

CS4243

Motion Tracking

25

Any difference in motion?

CS4243

Motion Tracking

26

Any difference in motion?

CS4243

Motion Tracking

27

Any difference in motion?

CS4243

Motion Tracking

28

Aperture Problem


Homogeneous region

 Ix

= Iy = It = 0.
 No change in local region.
 Cannot detect motion.

CS4243

Motion Tracking

29

Aperture Problem


Edge

 Ix

and Iy are zero along edge
 Cannot measure motion tangential to edge
 Ix and Iy are non-zero normal to edge
 Can measure motion normal to edge
 So, cannot measure actual motion

CS4243

Motion Tracking

30

Aperture Problem


Corner

 Ix

and Iy are non-zero in two perpendicular directions
 2 unknowns, 2 equations
 Can measure actual motion

CS4243

Motion Tracking

31

Lucas-Kanade Method


Consider two consecutive image frames I and J:
I

J
x+d
d
x

x

moves from x = (x, y)T to x + d.
 d = (u, v) T
 Object

CS4243

Motion Tracking

32



So,



Or



Due to noise, there’s an error at position x:



Sum error over small window W at position x:

Similar to
template matching
CS4243

weight
Motion Tracking

33



If E is small, patterns in I and J match well.



So, find d that minimises E:
E / d = 0, compute d that minimises E.
 First, expand I(x – d) by Taylor’s series expansion:
 Set

 Omit

higher order terms:

 Write

in matrix form:

Intensity gradient
CS4243

Motion Tracking

34

 Now,

error E at position x is:

 Now,

differentiate E with respect to d (exercise):

 Setting

E / d = 0 gives

b

the only
unknown

Z

CS4243

Motion Tracking

35

 So,

2



CS4243

we get

unknowns, 2 equations. Can solve for d = (u, v).

What happen to aperture problem?
Did it disappear?
Motion Tracking

36

Lucas-Kanade + Tomasi


Lucas-Kanade algorithm is often used with
Tomasi’s feature
 Apply

Tomasi’s method to detect good features.
 Apply LK method to compute d for each pixel.
 Accept d only for good features.

CS4243

Motion Tracking

37

Example

Can you spot tracking errors?

CS4243

Motion Tracking

38

Constraints


Math of LK tracker assumes d is small.



In implementation, W is also small.



LK tracker is good only for small displacement.

CS4243

Motion Tracking

39

How to handle large displacement?


What if we scale down images?
 Displacements

CS4243

are smaller!

Motion Tracking

40

Image Pyramid
Usually, smoothen
image with Gaussian
filter before scaling
down.

downsample

scaled image

input image
CS4243

Motion Tracking

41

LK Tracking with Image Pyramid

CS4243

Motion Tracking

42



Construct image pyramids.



Apply LK tracker to low-resolution images.



Propagate results to higher-resolution images.



Apply LK tracker to higher-resolution images.

CS4243

Motion Tracking

43

Example

Tracking results are more accurate.

CS4243

Motion Tracking

44

Summary


Efficient algorithm, no explicit search.



Has aperture problem;
track good features only



LK tracker can’t track large displacement.



Use LK + image pyramid for large displacement.

CS4243

Motion Tracking

45

Software


OpenCV supports LK and LK with pyramid.



[Bir] offers LK with Tomasi’s features & pyramid.

CS4243

Motion Tracking

46

Appendix


Calculation of Ix, Iy, It
 Use

finite difference method
 Forward difference
Ix = I(x  1, y, t)  I(x, y, t)
Iy = I(x, y  1, t)  I(x, y, t)
It = I(x, y, t  1)  I(x, y, t)
 Backward difference
Ix = I(x, y, t)  I(x  1, y, t)
Iy = I(x, y, t)  I(x, y  1, t)
It = I(x, y, t)  I(x, y, t  1)

CS4243

Motion Tracking

47

Further Readings


Lucas-Kanade tracking with pyramid:
[BK08] Chapter 10.



Optical flow: [Sze10] Section 8.4.



Hierarchical motion estimation (with image
pyramid): [Sze10] Section 8.1.1.

CS4243

Motion Tracking

48

References


[Bir] S. Birchfield. KLT: An implementation of the Kanade-LucasTomasi feature tracker. http://vision.stanford.edu/~birch/klt/.



[BK08] Bradski and Kaehler. Learning OpenCV: Computer Vision
with the OpenCV Library. O’Reilly, 2008.



[LK81] B. D. Lucas and T. Kanade. An iterative image registration
technique with an application to stereo vision. In Proceedings of 7th
International Joint Conference on Artificial Intelligence, pages 674–
679, 1981.



[ST94] J. Shi and C. Tomasi. Good features to track. In Proceedings
of IEEE Conference on Computer Vision and Pattern Recognition,
pages 593–600, 1994.



[Sze10] R. Szeliski. Computer Vision: Algorithms and Applications.
Springer, 2010.

CS4243

Motion Tracking

49

References


CS4243

[TK91] C. Tomasi and T. Kanade. Detection and tracking of point
features. Technical Report CMU-CS-91-132, School of Computer
Science, Carnegie Mellon University, 1991.

Motion Tracking

50


Slide 10

Leow Wee Kheng
CS4243 Computer Vision and Pattern Recognition

Motion Tracking

CS4243

Motion Tracking

1

Changes are everywhere!

CS4243

Motion Tracking

2

Illumination change

CS4243

Motion Tracking

3

Shape change

CS4243

Motion Tracking

4

Object motion

CS4243

Motion Tracking

5

Camera motion

CS4243

Motion Tracking

6

Object & camera motion

CS4243

Motion Tracking

7

Everything changes!

CS4243

Motion Tracking

8

Motion analysis is tough in general!

We focus on object / camera motion.

CS4243

Motion Tracking

9

Change Detection


Detects any change in two video frames.



Straightforward method:
 Compute

difference between corresponding pixels:



: intensity / colour at (x, y) in frame t.
> threshold, has large difference.

 If

CS4243

Motion Tracking

10

Any difference?

No

CS4243

Motion Tracking

11

Any difference?

Yes,
illumination
change

CS4243

Motion Tracking

12

Any difference?

Yes,
position change

CS4243

Motion Tracking

13

Change Detection


Can detect
 Illumination

change
 Position change
 Illumination and position change


But, cannot distinguish between them.



Need to detect and measure position change.

CS4243

Motion Tracking

14

Motion Tracking


Two approaches
 Feature-based
 Intensity

CS4243

gradient-based

Motion Tracking

15

Feature-based Motion Tracking


Look for distinct features that change positions.

 Eagle’s

wing tips change positions.
 Tree tops don’t change positions.
CS4243

Motion Tracking

16

Basic Ideas
1.

Look for distinct features in current frame.

2.

For each feature


Search for matching feature within neighbourhood
in next frame.
 Difference in positions  displacement.
 Velocity = displacement / time difference.

CS4243

Motion Tracking

17

Basic Ideas
displacement

feature area

search area

CS4243

Motion Tracking

18

What feature to use?


Harris corner



Tomasi’s feature



Feature descriptors: SIFT, SURF, GLOH, etc.



Others

CS4243

Motion Tracking

19

Summary


Simple algorithm.



Can be slow if search area is large.



Can constrain search area with prior knowledge.

CS4243

Motion Tracking

20

Gradient-based Motion Tracking


Two basic assumptions
 Intensity

changes smoothly with position.
 Intensity of object doesn’t change over time.


Suppose an object is in motion.
 Change

position (dx, dy) over time dt.
 Then, from 2nd assumption:
 Apply

CS4243

Taylor’s series expansion:

Motion Tracking

21

 Omit

higher order terms and divide by dt

 Denote

 Then,

 u,

v are unknown.
 2 unknowns, 1 equation: can’t solve!

CS4243

Motion Tracking

22

Any difference in motion?

CS4243

Motion Tracking

23

Any difference in motion?

CS4243

Motion Tracking

24

Any difference in motion?

CS4243

Motion Tracking

25

Any difference in motion?

CS4243

Motion Tracking

26

Any difference in motion?

CS4243

Motion Tracking

27

Any difference in motion?

CS4243

Motion Tracking

28

Aperture Problem


Homogeneous region

 Ix

= Iy = It = 0.
 No change in local region.
 Cannot detect motion.

CS4243

Motion Tracking

29

Aperture Problem


Edge

 Ix

and Iy are zero along edge
 Cannot measure motion tangential to edge
 Ix and Iy are non-zero normal to edge
 Can measure motion normal to edge
 So, cannot measure actual motion

CS4243

Motion Tracking

30

Aperture Problem


Corner

 Ix

and Iy are non-zero in two perpendicular directions
 2 unknowns, 2 equations
 Can measure actual motion

CS4243

Motion Tracking

31

Lucas-Kanade Method


Consider two consecutive image frames I and J:
I

J
x+d
d
x

x

moves from x = (x, y)T to x + d.
 d = (u, v) T
 Object

CS4243

Motion Tracking

32



So,



Or



Due to noise, there’s an error at position x:



Sum error over small window W at position x:

Similar to
template matching
CS4243

weight
Motion Tracking

33



If E is small, patterns in I and J match well.



So, find d that minimises E:
E / d = 0, compute d that minimises E.
 First, expand I(x – d) by Taylor’s series expansion:
 Set

 Omit

higher order terms:

 Write

in matrix form:

Intensity gradient
CS4243

Motion Tracking

34

 Now,

error E at position x is:

 Now,

differentiate E with respect to d (exercise):

 Setting

E / d = 0 gives

b

the only
unknown

Z

CS4243

Motion Tracking

35

 So,

2



CS4243

we get

unknowns, 2 equations. Can solve for d = (u, v).

What happen to aperture problem?
Did it disappear?
Motion Tracking

36

Lucas-Kanade + Tomasi


Lucas-Kanade algorithm is often used with
Tomasi’s feature
 Apply

Tomasi’s method to detect good features.
 Apply LK method to compute d for each pixel.
 Accept d only for good features.

CS4243

Motion Tracking

37

Example

Can you spot tracking errors?

CS4243

Motion Tracking

38

Constraints


Math of LK tracker assumes d is small.



In implementation, W is also small.



LK tracker is good only for small displacement.

CS4243

Motion Tracking

39

How to handle large displacement?


What if we scale down images?
 Displacements

CS4243

are smaller!

Motion Tracking

40

Image Pyramid
Usually, smoothen
image with Gaussian
filter before scaling
down.

downsample

scaled image

input image
CS4243

Motion Tracking

41

LK Tracking with Image Pyramid

CS4243

Motion Tracking

42



Construct image pyramids.



Apply LK tracker to low-resolution images.



Propagate results to higher-resolution images.



Apply LK tracker to higher-resolution images.

CS4243

Motion Tracking

43

Example

Tracking results are more accurate.

CS4243

Motion Tracking

44

Summary


Efficient algorithm, no explicit search.



Has aperture problem;
track good features only



LK tracker can’t track large displacement.



Use LK + image pyramid for large displacement.

CS4243

Motion Tracking

45

Software


OpenCV supports LK and LK with pyramid.



[Bir] offers LK with Tomasi’s features & pyramid.

CS4243

Motion Tracking

46

Appendix


Calculation of Ix, Iy, It
 Use

finite difference method
 Forward difference
Ix = I(x  1, y, t)  I(x, y, t)
Iy = I(x, y  1, t)  I(x, y, t)
It = I(x, y, t  1)  I(x, y, t)
 Backward difference
Ix = I(x, y, t)  I(x  1, y, t)
Iy = I(x, y, t)  I(x, y  1, t)
It = I(x, y, t)  I(x, y, t  1)

CS4243

Motion Tracking

47

Further Readings


Lucas-Kanade tracking with pyramid:
[BK08] Chapter 10.



Optical flow: [Sze10] Section 8.4.



Hierarchical motion estimation (with image
pyramid): [Sze10] Section 8.1.1.

CS4243

Motion Tracking

48

References


[Bir] S. Birchfield. KLT: An implementation of the Kanade-LucasTomasi feature tracker. http://vision.stanford.edu/~birch/klt/.



[BK08] Bradski and Kaehler. Learning OpenCV: Computer Vision
with the OpenCV Library. O’Reilly, 2008.



[LK81] B. D. Lucas and T. Kanade. An iterative image registration
technique with an application to stereo vision. In Proceedings of 7th
International Joint Conference on Artificial Intelligence, pages 674–
679, 1981.



[ST94] J. Shi and C. Tomasi. Good features to track. In Proceedings
of IEEE Conference on Computer Vision and Pattern Recognition,
pages 593–600, 1994.



[Sze10] R. Szeliski. Computer Vision: Algorithms and Applications.
Springer, 2010.

CS4243

Motion Tracking

49

References


CS4243

[TK91] C. Tomasi and T. Kanade. Detection and tracking of point
features. Technical Report CMU-CS-91-132, School of Computer
Science, Carnegie Mellon University, 1991.

Motion Tracking

50


Slide 11

Leow Wee Kheng
CS4243 Computer Vision and Pattern Recognition

Motion Tracking

CS4243

Motion Tracking

1

Changes are everywhere!

CS4243

Motion Tracking

2

Illumination change

CS4243

Motion Tracking

3

Shape change

CS4243

Motion Tracking

4

Object motion

CS4243

Motion Tracking

5

Camera motion

CS4243

Motion Tracking

6

Object & camera motion

CS4243

Motion Tracking

7

Everything changes!

CS4243

Motion Tracking

8

Motion analysis is tough in general!

We focus on object / camera motion.

CS4243

Motion Tracking

9

Change Detection


Detects any change in two video frames.



Straightforward method:
 Compute

difference between corresponding pixels:



: intensity / colour at (x, y) in frame t.
> threshold, has large difference.

 If

CS4243

Motion Tracking

10

Any difference?

No

CS4243

Motion Tracking

11

Any difference?

Yes,
illumination
change

CS4243

Motion Tracking

12

Any difference?

Yes,
position change

CS4243

Motion Tracking

13

Change Detection


Can detect
 Illumination

change
 Position change
 Illumination and position change


But, cannot distinguish between them.



Need to detect and measure position change.

CS4243

Motion Tracking

14

Motion Tracking


Two approaches
 Feature-based
 Intensity

CS4243

gradient-based

Motion Tracking

15

Feature-based Motion Tracking


Look for distinct features that change positions.

 Eagle’s

wing tips change positions.
 Tree tops don’t change positions.
CS4243

Motion Tracking

16

Basic Ideas
1.

Look for distinct features in current frame.

2.

For each feature


Search for matching feature within neighbourhood
in next frame.
 Difference in positions  displacement.
 Velocity = displacement / time difference.

CS4243

Motion Tracking

17

Basic Ideas
displacement

feature area

search area

CS4243

Motion Tracking

18

What feature to use?


Harris corner



Tomasi’s feature



Feature descriptors: SIFT, SURF, GLOH, etc.



Others

CS4243

Motion Tracking

19

Summary


Simple algorithm.



Can be slow if search area is large.



Can constrain search area with prior knowledge.

CS4243

Motion Tracking

20

Gradient-based Motion Tracking


Two basic assumptions
 Intensity

changes smoothly with position.
 Intensity of object doesn’t change over time.


Suppose an object is in motion.
 Change

position (dx, dy) over time dt.
 Then, from 2nd assumption:
 Apply

CS4243

Taylor’s series expansion:

Motion Tracking

21

 Omit

higher order terms and divide by dt

 Denote

 Then,

 u,

v are unknown.
 2 unknowns, 1 equation: can’t solve!

CS4243

Motion Tracking

22

Any difference in motion?

CS4243

Motion Tracking

23

Any difference in motion?

CS4243

Motion Tracking

24

Any difference in motion?

CS4243

Motion Tracking

25

Any difference in motion?

CS4243

Motion Tracking

26

Any difference in motion?

CS4243

Motion Tracking

27

Any difference in motion?

CS4243

Motion Tracking

28

Aperture Problem


Homogeneous region

 Ix

= Iy = It = 0.
 No change in local region.
 Cannot detect motion.

CS4243

Motion Tracking

29

Aperture Problem


Edge

 Ix

and Iy are zero along edge
 Cannot measure motion tangential to edge
 Ix and Iy are non-zero normal to edge
 Can measure motion normal to edge
 So, cannot measure actual motion

CS4243

Motion Tracking

30

Aperture Problem


Corner

 Ix

and Iy are non-zero in two perpendicular directions
 2 unknowns, 2 equations
 Can measure actual motion

CS4243

Motion Tracking

31

Lucas-Kanade Method


Consider two consecutive image frames I and J:
I

J
x+d
d
x

x

moves from x = (x, y)T to x + d.
 d = (u, v) T
 Object

CS4243

Motion Tracking

32



So,



Or



Due to noise, there’s an error at position x:



Sum error over small window W at position x:

Similar to
template matching
CS4243

weight
Motion Tracking

33



If E is small, patterns in I and J match well.



So, find d that minimises E:
E / d = 0, compute d that minimises E.
 First, expand I(x – d) by Taylor’s series expansion:
 Set

 Omit

higher order terms:

 Write

in matrix form:

Intensity gradient
CS4243

Motion Tracking

34

 Now,

error E at position x is:

 Now,

differentiate E with respect to d (exercise):

 Setting

E / d = 0 gives

b

the only
unknown

Z

CS4243

Motion Tracking

35

 So,

2



CS4243

we get

unknowns, 2 equations. Can solve for d = (u, v).

What happen to aperture problem?
Did it disappear?
Motion Tracking

36

Lucas-Kanade + Tomasi


Lucas-Kanade algorithm is often used with
Tomasi’s feature
 Apply

Tomasi’s method to detect good features.
 Apply LK method to compute d for each pixel.
 Accept d only for good features.

CS4243

Motion Tracking

37

Example

Can you spot tracking errors?

CS4243

Motion Tracking

38

Constraints


Math of LK tracker assumes d is small.



In implementation, W is also small.



LK tracker is good only for small displacement.

CS4243

Motion Tracking

39

How to handle large displacement?


What if we scale down images?
 Displacements

CS4243

are smaller!

Motion Tracking

40

Image Pyramid
Usually, smoothen
image with Gaussian
filter before scaling
down.

downsample

scaled image

input image
CS4243

Motion Tracking

41

LK Tracking with Image Pyramid

CS4243

Motion Tracking

42



Construct image pyramids.



Apply LK tracker to low-resolution images.



Propagate results to higher-resolution images.



Apply LK tracker to higher-resolution images.

CS4243

Motion Tracking

43

Example

Tracking results are more accurate.

CS4243

Motion Tracking

44

Summary


Efficient algorithm, no explicit search.



Has aperture problem;
track good features only



LK tracker can’t track large displacement.



Use LK + image pyramid for large displacement.

CS4243

Motion Tracking

45

Software


OpenCV supports LK and LK with pyramid.



[Bir] offers LK with Tomasi’s features & pyramid.

CS4243

Motion Tracking

46

Appendix


Calculation of Ix, Iy, It
 Use

finite difference method
 Forward difference
Ix = I(x  1, y, t)  I(x, y, t)
Iy = I(x, y  1, t)  I(x, y, t)
It = I(x, y, t  1)  I(x, y, t)
 Backward difference
Ix = I(x, y, t)  I(x  1, y, t)
Iy = I(x, y, t)  I(x, y  1, t)
It = I(x, y, t)  I(x, y, t  1)

CS4243

Motion Tracking

47

Further Readings


Lucas-Kanade tracking with pyramid:
[BK08] Chapter 10.



Optical flow: [Sze10] Section 8.4.



Hierarchical motion estimation (with image
pyramid): [Sze10] Section 8.1.1.

CS4243

Motion Tracking

48

References


[Bir] S. Birchfield. KLT: An implementation of the Kanade-LucasTomasi feature tracker. http://vision.stanford.edu/~birch/klt/.



[BK08] Bradski and Kaehler. Learning OpenCV: Computer Vision
with the OpenCV Library. O’Reilly, 2008.



[LK81] B. D. Lucas and T. Kanade. An iterative image registration
technique with an application to stereo vision. In Proceedings of 7th
International Joint Conference on Artificial Intelligence, pages 674–
679, 1981.



[ST94] J. Shi and C. Tomasi. Good features to track. In Proceedings
of IEEE Conference on Computer Vision and Pattern Recognition,
pages 593–600, 1994.



[Sze10] R. Szeliski. Computer Vision: Algorithms and Applications.
Springer, 2010.

CS4243

Motion Tracking

49

References


CS4243

[TK91] C. Tomasi and T. Kanade. Detection and tracking of point
features. Technical Report CMU-CS-91-132, School of Computer
Science, Carnegie Mellon University, 1991.

Motion Tracking

50


Slide 12

Leow Wee Kheng
CS4243 Computer Vision and Pattern Recognition

Motion Tracking

CS4243

Motion Tracking

1

Changes are everywhere!

CS4243

Motion Tracking

2

Illumination change

CS4243

Motion Tracking

3

Shape change

CS4243

Motion Tracking

4

Object motion

CS4243

Motion Tracking

5

Camera motion

CS4243

Motion Tracking

6

Object & camera motion

CS4243

Motion Tracking

7

Everything changes!

CS4243

Motion Tracking

8

Motion analysis is tough in general!

We focus on object / camera motion.

CS4243

Motion Tracking

9

Change Detection


Detects any change in two video frames.



Straightforward method:
 Compute

difference between corresponding pixels:



: intensity / colour at (x, y) in frame t.
> threshold, has large difference.

 If

CS4243

Motion Tracking

10

Any difference?

No

CS4243

Motion Tracking

11

Any difference?

Yes,
illumination
change

CS4243

Motion Tracking

12

Any difference?

Yes,
position change

CS4243

Motion Tracking

13

Change Detection


Can detect
 Illumination

change
 Position change
 Illumination and position change


But, cannot distinguish between them.



Need to detect and measure position change.

CS4243

Motion Tracking

14

Motion Tracking


Two approaches
 Feature-based
 Intensity

CS4243

gradient-based

Motion Tracking

15

Feature-based Motion Tracking


Look for distinct features that change positions.

 Eagle’s

wing tips change positions.
 Tree tops don’t change positions.
CS4243

Motion Tracking

16

Basic Ideas
1.

Look for distinct features in current frame.

2.

For each feature


Search for matching feature within neighbourhood
in next frame.
 Difference in positions  displacement.
 Velocity = displacement / time difference.

CS4243

Motion Tracking

17

Basic Ideas
displacement

feature area

search area

CS4243

Motion Tracking

18

What feature to use?


Harris corner



Tomasi’s feature



Feature descriptors: SIFT, SURF, GLOH, etc.



Others

CS4243

Motion Tracking

19

Summary


Simple algorithm.



Can be slow if search area is large.



Can constrain search area with prior knowledge.

CS4243

Motion Tracking

20

Gradient-based Motion Tracking


Two basic assumptions
 Intensity

changes smoothly with position.
 Intensity of object doesn’t change over time.


Suppose an object is in motion.
 Change

position (dx, dy) over time dt.
 Then, from 2nd assumption:
 Apply

CS4243

Taylor’s series expansion:

Motion Tracking

21

 Omit

higher order terms and divide by dt

 Denote

 Then,

 u,

v are unknown.
 2 unknowns, 1 equation: can’t solve!

CS4243

Motion Tracking

22

Any difference in motion?

CS4243

Motion Tracking

23

Any difference in motion?

CS4243

Motion Tracking

24

Any difference in motion?

CS4243

Motion Tracking

25

Any difference in motion?

CS4243

Motion Tracking

26

Any difference in motion?

CS4243

Motion Tracking

27

Any difference in motion?

CS4243

Motion Tracking

28

Aperture Problem


Homogeneous region

 Ix

= Iy = It = 0.
 No change in local region.
 Cannot detect motion.

CS4243

Motion Tracking

29

Aperture Problem


Edge

 Ix

and Iy are zero along edge
 Cannot measure motion tangential to edge
 Ix and Iy are non-zero normal to edge
 Can measure motion normal to edge
 So, cannot measure actual motion

CS4243

Motion Tracking

30

Aperture Problem


Corner

 Ix

and Iy are non-zero in two perpendicular directions
 2 unknowns, 2 equations
 Can measure actual motion

CS4243

Motion Tracking

31

Lucas-Kanade Method


Consider two consecutive image frames I and J:
I

J
x+d
d
x

x

moves from x = (x, y)T to x + d.
 d = (u, v) T
 Object

CS4243

Motion Tracking

32



So,



Or



Due to noise, there’s an error at position x:



Sum error over small window W at position x:

Similar to
template matching
CS4243

weight
Motion Tracking

33



If E is small, patterns in I and J match well.



So, find d that minimises E:
E / d = 0, compute d that minimises E.
 First, expand I(x – d) by Taylor’s series expansion:
 Set

 Omit

higher order terms:

 Write

in matrix form:

Intensity gradient
CS4243

Motion Tracking

34

 Now,

error E at position x is:

 Now,

differentiate E with respect to d (exercise):

 Setting

E / d = 0 gives

b

the only
unknown

Z

CS4243

Motion Tracking

35

 So,

2



CS4243

we get

unknowns, 2 equations. Can solve for d = (u, v).

What happen to aperture problem?
Did it disappear?
Motion Tracking

36

Lucas-Kanade + Tomasi


Lucas-Kanade algorithm is often used with
Tomasi’s feature
 Apply

Tomasi’s method to detect good features.
 Apply LK method to compute d for each pixel.
 Accept d only for good features.

CS4243

Motion Tracking

37

Example

Can you spot tracking errors?

CS4243

Motion Tracking

38

Constraints


Math of LK tracker assumes d is small.



In implementation, W is also small.



LK tracker is good only for small displacement.

CS4243

Motion Tracking

39

How to handle large displacement?


What if we scale down images?
 Displacements

CS4243

are smaller!

Motion Tracking

40

Image Pyramid
Usually, smoothen
image with Gaussian
filter before scaling
down.

downsample

scaled image

input image
CS4243

Motion Tracking

41

LK Tracking with Image Pyramid

CS4243

Motion Tracking

42



Construct image pyramids.



Apply LK tracker to low-resolution images.



Propagate results to higher-resolution images.



Apply LK tracker to higher-resolution images.

CS4243

Motion Tracking

43

Example

Tracking results are more accurate.

CS4243

Motion Tracking

44

Summary


Efficient algorithm, no explicit search.



Has aperture problem;
track good features only



LK tracker can’t track large displacement.



Use LK + image pyramid for large displacement.

CS4243

Motion Tracking

45

Software


OpenCV supports LK and LK with pyramid.



[Bir] offers LK with Tomasi’s features & pyramid.

CS4243

Motion Tracking

46

Appendix


Calculation of Ix, Iy, It
 Use

finite difference method
 Forward difference
Ix = I(x  1, y, t)  I(x, y, t)
Iy = I(x, y  1, t)  I(x, y, t)
It = I(x, y, t  1)  I(x, y, t)
 Backward difference
Ix = I(x, y, t)  I(x  1, y, t)
Iy = I(x, y, t)  I(x, y  1, t)
It = I(x, y, t)  I(x, y, t  1)

CS4243

Motion Tracking

47

Further Readings


Lucas-Kanade tracking with pyramid:
[BK08] Chapter 10.



Optical flow: [Sze10] Section 8.4.



Hierarchical motion estimation (with image
pyramid): [Sze10] Section 8.1.1.

CS4243

Motion Tracking

48

References


[Bir] S. Birchfield. KLT: An implementation of the Kanade-LucasTomasi feature tracker. http://vision.stanford.edu/~birch/klt/.



[BK08] Bradski and Kaehler. Learning OpenCV: Computer Vision
with the OpenCV Library. O’Reilly, 2008.



[LK81] B. D. Lucas and T. Kanade. An iterative image registration
technique with an application to stereo vision. In Proceedings of 7th
International Joint Conference on Artificial Intelligence, pages 674–
679, 1981.



[ST94] J. Shi and C. Tomasi. Good features to track. In Proceedings
of IEEE Conference on Computer Vision and Pattern Recognition,
pages 593–600, 1994.



[Sze10] R. Szeliski. Computer Vision: Algorithms and Applications.
Springer, 2010.

CS4243

Motion Tracking

49

References


CS4243

[TK91] C. Tomasi and T. Kanade. Detection and tracking of point
features. Technical Report CMU-CS-91-132, School of Computer
Science, Carnegie Mellon University, 1991.

Motion Tracking

50


Slide 13

Leow Wee Kheng
CS4243 Computer Vision and Pattern Recognition

Motion Tracking

CS4243

Motion Tracking

1

Changes are everywhere!

CS4243

Motion Tracking

2

Illumination change

CS4243

Motion Tracking

3

Shape change

CS4243

Motion Tracking

4

Object motion

CS4243

Motion Tracking

5

Camera motion

CS4243

Motion Tracking

6

Object & camera motion

CS4243

Motion Tracking

7

Everything changes!

CS4243

Motion Tracking

8

Motion analysis is tough in general!

We focus on object / camera motion.

CS4243

Motion Tracking

9

Change Detection


Detects any change in two video frames.



Straightforward method:
 Compute

difference between corresponding pixels:



: intensity / colour at (x, y) in frame t.
> threshold, has large difference.

 If

CS4243

Motion Tracking

10

Any difference?

No

CS4243

Motion Tracking

11

Any difference?

Yes,
illumination
change

CS4243

Motion Tracking

12

Any difference?

Yes,
position change

CS4243

Motion Tracking

13

Change Detection


Can detect
 Illumination

change
 Position change
 Illumination and position change


But, cannot distinguish between them.



Need to detect and measure position change.

CS4243

Motion Tracking

14

Motion Tracking


Two approaches
 Feature-based
 Intensity

CS4243

gradient-based

Motion Tracking

15

Feature-based Motion Tracking


Look for distinct features that change positions.

 Eagle’s

wing tips change positions.
 Tree tops don’t change positions.
CS4243

Motion Tracking

16

Basic Ideas
1.

Look for distinct features in current frame.

2.

For each feature


Search for matching feature within neighbourhood
in next frame.
 Difference in positions  displacement.
 Velocity = displacement / time difference.

CS4243

Motion Tracking

17

Basic Ideas
displacement

feature area

search area

CS4243

Motion Tracking

18

What feature to use?


Harris corner



Tomasi’s feature



Feature descriptors: SIFT, SURF, GLOH, etc.



Others

CS4243

Motion Tracking

19

Summary


Simple algorithm.



Can be slow if search area is large.



Can constrain search area with prior knowledge.

CS4243

Motion Tracking

20

Gradient-based Motion Tracking


Two basic assumptions
 Intensity

changes smoothly with position.
 Intensity of object doesn’t change over time.


Suppose an object is in motion.
 Change

position (dx, dy) over time dt.
 Then, from 2nd assumption:
 Apply

CS4243

Taylor’s series expansion:

Motion Tracking

21

 Omit

higher order terms and divide by dt

 Denote

 Then,

 u,

v are unknown.
 2 unknowns, 1 equation: can’t solve!

CS4243

Motion Tracking

22

Any difference in motion?

CS4243

Motion Tracking

23

Any difference in motion?

CS4243

Motion Tracking

24

Any difference in motion?

CS4243

Motion Tracking

25

Any difference in motion?

CS4243

Motion Tracking

26

Any difference in motion?

CS4243

Motion Tracking

27

Any difference in motion?

CS4243

Motion Tracking

28

Aperture Problem


Homogeneous region

 Ix

= Iy = It = 0.
 No change in local region.
 Cannot detect motion.

CS4243

Motion Tracking

29

Aperture Problem


Edge

 Ix

and Iy are zero along edge
 Cannot measure motion tangential to edge
 Ix and Iy are non-zero normal to edge
 Can measure motion normal to edge
 So, cannot measure actual motion

CS4243

Motion Tracking

30

Aperture Problem


Corner

 Ix

and Iy are non-zero in two perpendicular directions
 2 unknowns, 2 equations
 Can measure actual motion

CS4243

Motion Tracking

31

Lucas-Kanade Method


Consider two consecutive image frames I and J:
I

J
x+d
d
x

x

moves from x = (x, y)T to x + d.
 d = (u, v) T
 Object

CS4243

Motion Tracking

32



So,



Or



Due to noise, there’s an error at position x:



Sum error over small window W at position x:

Similar to
template matching
CS4243

weight
Motion Tracking

33



If E is small, patterns in I and J match well.



So, find d that minimises E:
E / d = 0, compute d that minimises E.
 First, expand I(x – d) by Taylor’s series expansion:
 Set

 Omit

higher order terms:

 Write

in matrix form:

Intensity gradient
CS4243

Motion Tracking

34

 Now,

error E at position x is:

 Now,

differentiate E with respect to d (exercise):

 Setting

E / d = 0 gives

b

the only
unknown

Z

CS4243

Motion Tracking

35

 So,

2



CS4243

we get

unknowns, 2 equations. Can solve for d = (u, v).

What happen to aperture problem?
Did it disappear?
Motion Tracking

36

Lucas-Kanade + Tomasi


Lucas-Kanade algorithm is often used with
Tomasi’s feature
 Apply

Tomasi’s method to detect good features.
 Apply LK method to compute d for each pixel.
 Accept d only for good features.

CS4243

Motion Tracking

37

Example

Can you spot tracking errors?

CS4243

Motion Tracking

38

Constraints


Math of LK tracker assumes d is small.



In implementation, W is also small.



LK tracker is good only for small displacement.

CS4243

Motion Tracking

39

How to handle large displacement?


What if we scale down images?
 Displacements

CS4243

are smaller!

Motion Tracking

40

Image Pyramid
Usually, smoothen
image with Gaussian
filter before scaling
down.

downsample

scaled image

input image
CS4243

Motion Tracking

41

LK Tracking with Image Pyramid

CS4243

Motion Tracking

42



Construct image pyramids.



Apply LK tracker to low-resolution images.



Propagate results to higher-resolution images.



Apply LK tracker to higher-resolution images.

CS4243

Motion Tracking

43

Example

Tracking results are more accurate.

CS4243

Motion Tracking

44

Summary


Efficient algorithm, no explicit search.



Has aperture problem;
track good features only



LK tracker can’t track large displacement.



Use LK + image pyramid for large displacement.

CS4243

Motion Tracking

45

Software


OpenCV supports LK and LK with pyramid.



[Bir] offers LK with Tomasi’s features & pyramid.

CS4243

Motion Tracking

46

Appendix


Calculation of Ix, Iy, It
 Use

finite difference method
 Forward difference
Ix = I(x  1, y, t)  I(x, y, t)
Iy = I(x, y  1, t)  I(x, y, t)
It = I(x, y, t  1)  I(x, y, t)
 Backward difference
Ix = I(x, y, t)  I(x  1, y, t)
Iy = I(x, y, t)  I(x, y  1, t)
It = I(x, y, t)  I(x, y, t  1)

CS4243

Motion Tracking

47

Further Readings


Lucas-Kanade tracking with pyramid:
[BK08] Chapter 10.



Optical flow: [Sze10] Section 8.4.



Hierarchical motion estimation (with image
pyramid): [Sze10] Section 8.1.1.

CS4243

Motion Tracking

48

References


[Bir] S. Birchfield. KLT: An implementation of the Kanade-LucasTomasi feature tracker. http://vision.stanford.edu/~birch/klt/.



[BK08] Bradski and Kaehler. Learning OpenCV: Computer Vision
with the OpenCV Library. O’Reilly, 2008.



[LK81] B. D. Lucas and T. Kanade. An iterative image registration
technique with an application to stereo vision. In Proceedings of 7th
International Joint Conference on Artificial Intelligence, pages 674–
679, 1981.



[ST94] J. Shi and C. Tomasi. Good features to track. In Proceedings
of IEEE Conference on Computer Vision and Pattern Recognition,
pages 593–600, 1994.



[Sze10] R. Szeliski. Computer Vision: Algorithms and Applications.
Springer, 2010.

CS4243

Motion Tracking

49

References


CS4243

[TK91] C. Tomasi and T. Kanade. Detection and tracking of point
features. Technical Report CMU-CS-91-132, School of Computer
Science, Carnegie Mellon University, 1991.

Motion Tracking

50


Slide 14

Leow Wee Kheng
CS4243 Computer Vision and Pattern Recognition

Motion Tracking

CS4243

Motion Tracking

1

Changes are everywhere!

CS4243

Motion Tracking

2

Illumination change

CS4243

Motion Tracking

3

Shape change

CS4243

Motion Tracking

4

Object motion

CS4243

Motion Tracking

5

Camera motion

CS4243

Motion Tracking

6

Object & camera motion

CS4243

Motion Tracking

7

Everything changes!

CS4243

Motion Tracking

8

Motion analysis is tough in general!

We focus on object / camera motion.

CS4243

Motion Tracking

9

Change Detection


Detects any change in two video frames.



Straightforward method:
 Compute

difference between corresponding pixels:



: intensity / colour at (x, y) in frame t.
> threshold, has large difference.

 If

CS4243

Motion Tracking

10

Any difference?

No

CS4243

Motion Tracking

11

Any difference?

Yes,
illumination
change

CS4243

Motion Tracking

12

Any difference?

Yes,
position change

CS4243

Motion Tracking

13

Change Detection


Can detect
 Illumination

change
 Position change
 Illumination and position change


But, cannot distinguish between them.



Need to detect and measure position change.

CS4243

Motion Tracking

14

Motion Tracking


Two approaches
 Feature-based
 Intensity

CS4243

gradient-based

Motion Tracking

15

Feature-based Motion Tracking


Look for distinct features that change positions.

 Eagle’s

wing tips change positions.
 Tree tops don’t change positions.
CS4243

Motion Tracking

16

Basic Ideas
1.

Look for distinct features in current frame.

2.

For each feature


Search for matching feature within neighbourhood
in next frame.
 Difference in positions  displacement.
 Velocity = displacement / time difference.

CS4243

Motion Tracking

17

Basic Ideas
displacement

feature area

search area

CS4243

Motion Tracking

18

What feature to use?


Harris corner



Tomasi’s feature



Feature descriptors: SIFT, SURF, GLOH, etc.



Others

CS4243

Motion Tracking

19

Summary


Simple algorithm.



Can be slow if search area is large.



Can constrain search area with prior knowledge.

CS4243

Motion Tracking

20

Gradient-based Motion Tracking


Two basic assumptions
 Intensity

changes smoothly with position.
 Intensity of object doesn’t change over time.


Suppose an object is in motion.
 Change

position (dx, dy) over time dt.
 Then, from 2nd assumption:
 Apply

CS4243

Taylor’s series expansion:

Motion Tracking

21

 Omit

higher order terms and divide by dt

 Denote

 Then,

 u,

v are unknown.
 2 unknowns, 1 equation: can’t solve!

CS4243

Motion Tracking

22

Any difference in motion?

CS4243

Motion Tracking

23

Any difference in motion?

CS4243

Motion Tracking

24

Any difference in motion?

CS4243

Motion Tracking

25

Any difference in motion?

CS4243

Motion Tracking

26

Any difference in motion?

CS4243

Motion Tracking

27

Any difference in motion?

CS4243

Motion Tracking

28

Aperture Problem


Homogeneous region

 Ix

= Iy = It = 0.
 No change in local region.
 Cannot detect motion.

CS4243

Motion Tracking

29

Aperture Problem


Edge

 Ix

and Iy are zero along edge
 Cannot measure motion tangential to edge
 Ix and Iy are non-zero normal to edge
 Can measure motion normal to edge
 So, cannot measure actual motion

CS4243

Motion Tracking

30

Aperture Problem


Corner

 Ix

and Iy are non-zero in two perpendicular directions
 2 unknowns, 2 equations
 Can measure actual motion

CS4243

Motion Tracking

31

Lucas-Kanade Method


Consider two consecutive image frames I and J:
I

J
x+d
d
x

x

moves from x = (x, y)T to x + d.
 d = (u, v) T
 Object

CS4243

Motion Tracking

32



So,



Or



Due to noise, there’s an error at position x:



Sum error over small window W at position x:

Similar to
template matching
CS4243

weight
Motion Tracking

33



If E is small, patterns in I and J match well.



So, find d that minimises E:
E / d = 0, compute d that minimises E.
 First, expand I(x – d) by Taylor’s series expansion:
 Set

 Omit

higher order terms:

 Write

in matrix form:

Intensity gradient
CS4243

Motion Tracking

34

 Now,

error E at position x is:

 Now,

differentiate E with respect to d (exercise):

 Setting

E / d = 0 gives

b

the only
unknown

Z

CS4243

Motion Tracking

35

 So,

2



CS4243

we get

unknowns, 2 equations. Can solve for d = (u, v).

What happen to aperture problem?
Did it disappear?
Motion Tracking

36

Lucas-Kanade + Tomasi


Lucas-Kanade algorithm is often used with
Tomasi’s feature
 Apply

Tomasi’s method to detect good features.
 Apply LK method to compute d for each pixel.
 Accept d only for good features.

CS4243

Motion Tracking

37

Example

Can you spot tracking errors?

CS4243

Motion Tracking

38

Constraints


Math of LK tracker assumes d is small.



In implementation, W is also small.



LK tracker is good only for small displacement.

CS4243

Motion Tracking

39

How to handle large displacement?


What if we scale down images?
 Displacements

CS4243

are smaller!

Motion Tracking

40

Image Pyramid
Usually, smoothen
image with Gaussian
filter before scaling
down.

downsample

scaled image

input image
CS4243

Motion Tracking

41

LK Tracking with Image Pyramid

CS4243

Motion Tracking

42



Construct image pyramids.



Apply LK tracker to low-resolution images.



Propagate results to higher-resolution images.



Apply LK tracker to higher-resolution images.

CS4243

Motion Tracking

43

Example

Tracking results are more accurate.

CS4243

Motion Tracking

44

Summary


Efficient algorithm, no explicit search.



Has aperture problem;
track good features only



LK tracker can’t track large displacement.



Use LK + image pyramid for large displacement.

CS4243

Motion Tracking

45

Software


OpenCV supports LK and LK with pyramid.



[Bir] offers LK with Tomasi’s features & pyramid.

CS4243

Motion Tracking

46

Appendix


Calculation of Ix, Iy, It
 Use

finite difference method
 Forward difference
Ix = I(x  1, y, t)  I(x, y, t)
Iy = I(x, y  1, t)  I(x, y, t)
It = I(x, y, t  1)  I(x, y, t)
 Backward difference
Ix = I(x, y, t)  I(x  1, y, t)
Iy = I(x, y, t)  I(x, y  1, t)
It = I(x, y, t)  I(x, y, t  1)

CS4243

Motion Tracking

47

Further Readings


Lucas-Kanade tracking with pyramid:
[BK08] Chapter 10.



Optical flow: [Sze10] Section 8.4.



Hierarchical motion estimation (with image
pyramid): [Sze10] Section 8.1.1.

CS4243

Motion Tracking

48

References


[Bir] S. Birchfield. KLT: An implementation of the Kanade-LucasTomasi feature tracker. http://vision.stanford.edu/~birch/klt/.



[BK08] Bradski and Kaehler. Learning OpenCV: Computer Vision
with the OpenCV Library. O’Reilly, 2008.



[LK81] B. D. Lucas and T. Kanade. An iterative image registration
technique with an application to stereo vision. In Proceedings of 7th
International Joint Conference on Artificial Intelligence, pages 674–
679, 1981.



[ST94] J. Shi and C. Tomasi. Good features to track. In Proceedings
of IEEE Conference on Computer Vision and Pattern Recognition,
pages 593–600, 1994.



[Sze10] R. Szeliski. Computer Vision: Algorithms and Applications.
Springer, 2010.

CS4243

Motion Tracking

49

References


CS4243

[TK91] C. Tomasi and T. Kanade. Detection and tracking of point
features. Technical Report CMU-CS-91-132, School of Computer
Science, Carnegie Mellon University, 1991.

Motion Tracking

50


Slide 15

Leow Wee Kheng
CS4243 Computer Vision and Pattern Recognition

Motion Tracking

CS4243

Motion Tracking

1

Changes are everywhere!

CS4243

Motion Tracking

2

Illumination change

CS4243

Motion Tracking

3

Shape change

CS4243

Motion Tracking

4

Object motion

CS4243

Motion Tracking

5

Camera motion

CS4243

Motion Tracking

6

Object & camera motion

CS4243

Motion Tracking

7

Everything changes!

CS4243

Motion Tracking

8

Motion analysis is tough in general!

We focus on object / camera motion.

CS4243

Motion Tracking

9

Change Detection


Detects any change in two video frames.



Straightforward method:
 Compute

difference between corresponding pixels:



: intensity / colour at (x, y) in frame t.
> threshold, has large difference.

 If

CS4243

Motion Tracking

10

Any difference?

No

CS4243

Motion Tracking

11

Any difference?

Yes,
illumination
change

CS4243

Motion Tracking

12

Any difference?

Yes,
position change

CS4243

Motion Tracking

13

Change Detection


Can detect
 Illumination

change
 Position change
 Illumination and position change


But, cannot distinguish between them.



Need to detect and measure position change.

CS4243

Motion Tracking

14

Motion Tracking


Two approaches
 Feature-based
 Intensity

CS4243

gradient-based

Motion Tracking

15

Feature-based Motion Tracking


Look for distinct features that change positions.

 Eagle’s

wing tips change positions.
 Tree tops don’t change positions.
CS4243

Motion Tracking

16

Basic Ideas
1.

Look for distinct features in current frame.

2.

For each feature


Search for matching feature within neighbourhood
in next frame.
 Difference in positions  displacement.
 Velocity = displacement / time difference.

CS4243

Motion Tracking

17

Basic Ideas
displacement

feature area

search area

CS4243

Motion Tracking

18

What feature to use?


Harris corner



Tomasi’s feature



Feature descriptors: SIFT, SURF, GLOH, etc.



Others

CS4243

Motion Tracking

19

Summary


Simple algorithm.



Can be slow if search area is large.



Can constrain search area with prior knowledge.

CS4243

Motion Tracking

20

Gradient-based Motion Tracking


Two basic assumptions
 Intensity

changes smoothly with position.
 Intensity of object doesn’t change over time.


Suppose an object is in motion.
 Change

position (dx, dy) over time dt.
 Then, from 2nd assumption:
 Apply

CS4243

Taylor’s series expansion:

Motion Tracking

21

 Omit

higher order terms and divide by dt

 Denote

 Then,

 u,

v are unknown.
 2 unknowns, 1 equation: can’t solve!

CS4243

Motion Tracking

22

Any difference in motion?

CS4243

Motion Tracking

23

Any difference in motion?

CS4243

Motion Tracking

24

Any difference in motion?

CS4243

Motion Tracking

25

Any difference in motion?

CS4243

Motion Tracking

26

Any difference in motion?

CS4243

Motion Tracking

27

Any difference in motion?

CS4243

Motion Tracking

28

Aperture Problem


Homogeneous region

 Ix

= Iy = It = 0.
 No change in local region.
 Cannot detect motion.

CS4243

Motion Tracking

29

Aperture Problem


Edge

 Ix

and Iy are zero along edge
 Cannot measure motion tangential to edge
 Ix and Iy are non-zero normal to edge
 Can measure motion normal to edge
 So, cannot measure actual motion

CS4243

Motion Tracking

30

Aperture Problem


Corner

 Ix

and Iy are non-zero in two perpendicular directions
 2 unknowns, 2 equations
 Can measure actual motion

CS4243

Motion Tracking

31

Lucas-Kanade Method


Consider two consecutive image frames I and J:
I

J
x+d
d
x

x

moves from x = (x, y)T to x + d.
 d = (u, v) T
 Object

CS4243

Motion Tracking

32



So,



Or



Due to noise, there’s an error at position x:



Sum error over small window W at position x:

Similar to
template matching
CS4243

weight
Motion Tracking

33



If E is small, patterns in I and J match well.



So, find d that minimises E:
E / d = 0, compute d that minimises E.
 First, expand I(x – d) by Taylor’s series expansion:
 Set

 Omit

higher order terms:

 Write

in matrix form:

Intensity gradient
CS4243

Motion Tracking

34

 Now,

error E at position x is:

 Now,

differentiate E with respect to d (exercise):

 Setting

E / d = 0 gives

b

the only
unknown

Z

CS4243

Motion Tracking

35

 So,

2



CS4243

we get

unknowns, 2 equations. Can solve for d = (u, v).

What happen to aperture problem?
Did it disappear?
Motion Tracking

36

Lucas-Kanade + Tomasi


Lucas-Kanade algorithm is often used with
Tomasi’s feature
 Apply

Tomasi’s method to detect good features.
 Apply LK method to compute d for each pixel.
 Accept d only for good features.

CS4243

Motion Tracking

37

Example

Can you spot tracking errors?

CS4243

Motion Tracking

38

Constraints


Math of LK tracker assumes d is small.



In implementation, W is also small.



LK tracker is good only for small displacement.

CS4243

Motion Tracking

39

How to handle large displacement?


What if we scale down images?
 Displacements

CS4243

are smaller!

Motion Tracking

40

Image Pyramid
Usually, smoothen
image with Gaussian
filter before scaling
down.

downsample

scaled image

input image
CS4243

Motion Tracking

41

LK Tracking with Image Pyramid

CS4243

Motion Tracking

42



Construct image pyramids.



Apply LK tracker to low-resolution images.



Propagate results to higher-resolution images.



Apply LK tracker to higher-resolution images.

CS4243

Motion Tracking

43

Example

Tracking results are more accurate.

CS4243

Motion Tracking

44

Summary


Efficient algorithm, no explicit search.



Has aperture problem;
track good features only



LK tracker can’t track large displacement.



Use LK + image pyramid for large displacement.

CS4243

Motion Tracking

45

Software


OpenCV supports LK and LK with pyramid.



[Bir] offers LK with Tomasi’s features & pyramid.

CS4243

Motion Tracking

46

Appendix


Calculation of Ix, Iy, It
 Use

finite difference method
 Forward difference
Ix = I(x  1, y, t)  I(x, y, t)
Iy = I(x, y  1, t)  I(x, y, t)
It = I(x, y, t  1)  I(x, y, t)
 Backward difference
Ix = I(x, y, t)  I(x  1, y, t)
Iy = I(x, y, t)  I(x, y  1, t)
It = I(x, y, t)  I(x, y, t  1)

CS4243

Motion Tracking

47

Further Readings


Lucas-Kanade tracking with pyramid:
[BK08] Chapter 10.



Optical flow: [Sze10] Section 8.4.



Hierarchical motion estimation (with image
pyramid): [Sze10] Section 8.1.1.

CS4243

Motion Tracking

48

References


[Bir] S. Birchfield. KLT: An implementation of the Kanade-LucasTomasi feature tracker. http://vision.stanford.edu/~birch/klt/.



[BK08] Bradski and Kaehler. Learning OpenCV: Computer Vision
with the OpenCV Library. O’Reilly, 2008.



[LK81] B. D. Lucas and T. Kanade. An iterative image registration
technique with an application to stereo vision. In Proceedings of 7th
International Joint Conference on Artificial Intelligence, pages 674–
679, 1981.



[ST94] J. Shi and C. Tomasi. Good features to track. In Proceedings
of IEEE Conference on Computer Vision and Pattern Recognition,
pages 593–600, 1994.



[Sze10] R. Szeliski. Computer Vision: Algorithms and Applications.
Springer, 2010.

CS4243

Motion Tracking

49

References


CS4243

[TK91] C. Tomasi and T. Kanade. Detection and tracking of point
features. Technical Report CMU-CS-91-132, School of Computer
Science, Carnegie Mellon University, 1991.

Motion Tracking

50


Slide 16

Leow Wee Kheng
CS4243 Computer Vision and Pattern Recognition

Motion Tracking

CS4243

Motion Tracking

1

Changes are everywhere!

CS4243

Motion Tracking

2

Illumination change

CS4243

Motion Tracking

3

Shape change

CS4243

Motion Tracking

4

Object motion

CS4243

Motion Tracking

5

Camera motion

CS4243

Motion Tracking

6

Object & camera motion

CS4243

Motion Tracking

7

Everything changes!

CS4243

Motion Tracking

8

Motion analysis is tough in general!

We focus on object / camera motion.

CS4243

Motion Tracking

9

Change Detection


Detects any change in two video frames.



Straightforward method:
 Compute

difference between corresponding pixels:



: intensity / colour at (x, y) in frame t.
> threshold, has large difference.

 If

CS4243

Motion Tracking

10

Any difference?

No

CS4243

Motion Tracking

11

Any difference?

Yes,
illumination
change

CS4243

Motion Tracking

12

Any difference?

Yes,
position change

CS4243

Motion Tracking

13

Change Detection


Can detect
 Illumination

change
 Position change
 Illumination and position change


But, cannot distinguish between them.



Need to detect and measure position change.

CS4243

Motion Tracking

14

Motion Tracking


Two approaches
 Feature-based
 Intensity

CS4243

gradient-based

Motion Tracking

15

Feature-based Motion Tracking


Look for distinct features that change positions.

 Eagle’s

wing tips change positions.
 Tree tops don’t change positions.
CS4243

Motion Tracking

16

Basic Ideas
1.

Look for distinct features in current frame.

2.

For each feature


Search for matching feature within neighbourhood
in next frame.
 Difference in positions  displacement.
 Velocity = displacement / time difference.

CS4243

Motion Tracking

17

Basic Ideas
displacement

feature area

search area

CS4243

Motion Tracking

18

What feature to use?


Harris corner



Tomasi’s feature



Feature descriptors: SIFT, SURF, GLOH, etc.



Others

CS4243

Motion Tracking

19

Summary


Simple algorithm.



Can be slow if search area is large.



Can constrain search area with prior knowledge.

CS4243

Motion Tracking

20

Gradient-based Motion Tracking


Two basic assumptions
 Intensity

changes smoothly with position.
 Intensity of object doesn’t change over time.


Suppose an object is in motion.
 Change

position (dx, dy) over time dt.
 Then, from 2nd assumption:
 Apply

CS4243

Taylor’s series expansion:

Motion Tracking

21

 Omit

higher order terms and divide by dt

 Denote

 Then,

 u,

v are unknown.
 2 unknowns, 1 equation: can’t solve!

CS4243

Motion Tracking

22

Any difference in motion?

CS4243

Motion Tracking

23

Any difference in motion?

CS4243

Motion Tracking

24

Any difference in motion?

CS4243

Motion Tracking

25

Any difference in motion?

CS4243

Motion Tracking

26

Any difference in motion?

CS4243

Motion Tracking

27

Any difference in motion?

CS4243

Motion Tracking

28

Aperture Problem


Homogeneous region

 Ix

= Iy = It = 0.
 No change in local region.
 Cannot detect motion.

CS4243

Motion Tracking

29

Aperture Problem


Edge

 Ix

and Iy are zero along edge
 Cannot measure motion tangential to edge
 Ix and Iy are non-zero normal to edge
 Can measure motion normal to edge
 So, cannot measure actual motion

CS4243

Motion Tracking

30

Aperture Problem


Corner

 Ix

and Iy are non-zero in two perpendicular directions
 2 unknowns, 2 equations
 Can measure actual motion

CS4243

Motion Tracking

31

Lucas-Kanade Method


Consider two consecutive image frames I and J:
I

J
x+d
d
x

x

moves from x = (x, y)T to x + d.
 d = (u, v) T
 Object

CS4243

Motion Tracking

32



So,



Or



Due to noise, there’s an error at position x:



Sum error over small window W at position x:

Similar to
template matching
CS4243

weight
Motion Tracking

33



If E is small, patterns in I and J match well.



So, find d that minimises E:
E / d = 0, compute d that minimises E.
 First, expand I(x – d) by Taylor’s series expansion:
 Set

 Omit

higher order terms:

 Write

in matrix form:

Intensity gradient
CS4243

Motion Tracking

34

 Now,

error E at position x is:

 Now,

differentiate E with respect to d (exercise):

 Setting

E / d = 0 gives

b

the only
unknown

Z

CS4243

Motion Tracking

35

 So,

2



CS4243

we get

unknowns, 2 equations. Can solve for d = (u, v).

What happen to aperture problem?
Did it disappear?
Motion Tracking

36

Lucas-Kanade + Tomasi


Lucas-Kanade algorithm is often used with
Tomasi’s feature
 Apply

Tomasi’s method to detect good features.
 Apply LK method to compute d for each pixel.
 Accept d only for good features.

CS4243

Motion Tracking

37

Example

Can you spot tracking errors?

CS4243

Motion Tracking

38

Constraints


Math of LK tracker assumes d is small.



In implementation, W is also small.



LK tracker is good only for small displacement.

CS4243

Motion Tracking

39

How to handle large displacement?


What if we scale down images?
 Displacements

CS4243

are smaller!

Motion Tracking

40

Image Pyramid
Usually, smoothen
image with Gaussian
filter before scaling
down.

downsample

scaled image

input image
CS4243

Motion Tracking

41

LK Tracking with Image Pyramid

CS4243

Motion Tracking

42



Construct image pyramids.



Apply LK tracker to low-resolution images.



Propagate results to higher-resolution images.



Apply LK tracker to higher-resolution images.

CS4243

Motion Tracking

43

Example

Tracking results are more accurate.

CS4243

Motion Tracking

44

Summary


Efficient algorithm, no explicit search.



Has aperture problem;
track good features only



LK tracker can’t track large displacement.



Use LK + image pyramid for large displacement.

CS4243

Motion Tracking

45

Software


OpenCV supports LK and LK with pyramid.



[Bir] offers LK with Tomasi’s features & pyramid.

CS4243

Motion Tracking

46

Appendix


Calculation of Ix, Iy, It
 Use

finite difference method
 Forward difference
Ix = I(x  1, y, t)  I(x, y, t)
Iy = I(x, y  1, t)  I(x, y, t)
It = I(x, y, t  1)  I(x, y, t)
 Backward difference
Ix = I(x, y, t)  I(x  1, y, t)
Iy = I(x, y, t)  I(x, y  1, t)
It = I(x, y, t)  I(x, y, t  1)

CS4243

Motion Tracking

47

Further Readings


Lucas-Kanade tracking with pyramid:
[BK08] Chapter 10.



Optical flow: [Sze10] Section 8.4.



Hierarchical motion estimation (with image
pyramid): [Sze10] Section 8.1.1.

CS4243

Motion Tracking

48

References


[Bir] S. Birchfield. KLT: An implementation of the Kanade-LucasTomasi feature tracker. http://vision.stanford.edu/~birch/klt/.



[BK08] Bradski and Kaehler. Learning OpenCV: Computer Vision
with the OpenCV Library. O’Reilly, 2008.



[LK81] B. D. Lucas and T. Kanade. An iterative image registration
technique with an application to stereo vision. In Proceedings of 7th
International Joint Conference on Artificial Intelligence, pages 674–
679, 1981.



[ST94] J. Shi and C. Tomasi. Good features to track. In Proceedings
of IEEE Conference on Computer Vision and Pattern Recognition,
pages 593–600, 1994.



[Sze10] R. Szeliski. Computer Vision: Algorithms and Applications.
Springer, 2010.

CS4243

Motion Tracking

49

References


CS4243

[TK91] C. Tomasi and T. Kanade. Detection and tracking of point
features. Technical Report CMU-CS-91-132, School of Computer
Science, Carnegie Mellon University, 1991.

Motion Tracking

50


Slide 17

Leow Wee Kheng
CS4243 Computer Vision and Pattern Recognition

Motion Tracking

CS4243

Motion Tracking

1

Changes are everywhere!

CS4243

Motion Tracking

2

Illumination change

CS4243

Motion Tracking

3

Shape change

CS4243

Motion Tracking

4

Object motion

CS4243

Motion Tracking

5

Camera motion

CS4243

Motion Tracking

6

Object & camera motion

CS4243

Motion Tracking

7

Everything changes!

CS4243

Motion Tracking

8

Motion analysis is tough in general!

We focus on object / camera motion.

CS4243

Motion Tracking

9

Change Detection


Detects any change in two video frames.



Straightforward method:
 Compute

difference between corresponding pixels:



: intensity / colour at (x, y) in frame t.
> threshold, has large difference.

 If

CS4243

Motion Tracking

10

Any difference?

No

CS4243

Motion Tracking

11

Any difference?

Yes,
illumination
change

CS4243

Motion Tracking

12

Any difference?

Yes,
position change

CS4243

Motion Tracking

13

Change Detection


Can detect
 Illumination

change
 Position change
 Illumination and position change


But, cannot distinguish between them.



Need to detect and measure position change.

CS4243

Motion Tracking

14

Motion Tracking


Two approaches
 Feature-based
 Intensity

CS4243

gradient-based

Motion Tracking

15

Feature-based Motion Tracking


Look for distinct features that change positions.

 Eagle’s

wing tips change positions.
 Tree tops don’t change positions.
CS4243

Motion Tracking

16

Basic Ideas
1.

Look for distinct features in current frame.

2.

For each feature


Search for matching feature within neighbourhood
in next frame.
 Difference in positions  displacement.
 Velocity = displacement / time difference.

CS4243

Motion Tracking

17

Basic Ideas
displacement

feature area

search area

CS4243

Motion Tracking

18

What feature to use?


Harris corner



Tomasi’s feature



Feature descriptors: SIFT, SURF, GLOH, etc.



Others

CS4243

Motion Tracking

19

Summary


Simple algorithm.



Can be slow if search area is large.



Can constrain search area with prior knowledge.

CS4243

Motion Tracking

20

Gradient-based Motion Tracking


Two basic assumptions
 Intensity

changes smoothly with position.
 Intensity of object doesn’t change over time.


Suppose an object is in motion.
 Change

position (dx, dy) over time dt.
 Then, from 2nd assumption:
 Apply

CS4243

Taylor’s series expansion:

Motion Tracking

21

 Omit

higher order terms and divide by dt

 Denote

 Then,

 u,

v are unknown.
 2 unknowns, 1 equation: can’t solve!

CS4243

Motion Tracking

22

Any difference in motion?

CS4243

Motion Tracking

23

Any difference in motion?

CS4243

Motion Tracking

24

Any difference in motion?

CS4243

Motion Tracking

25

Any difference in motion?

CS4243

Motion Tracking

26

Any difference in motion?

CS4243

Motion Tracking

27

Any difference in motion?

CS4243

Motion Tracking

28

Aperture Problem


Homogeneous region

 Ix

= Iy = It = 0.
 No change in local region.
 Cannot detect motion.

CS4243

Motion Tracking

29

Aperture Problem


Edge

 Ix

and Iy are zero along edge
 Cannot measure motion tangential to edge
 Ix and Iy are non-zero normal to edge
 Can measure motion normal to edge
 So, cannot measure actual motion

CS4243

Motion Tracking

30

Aperture Problem


Corner

 Ix

and Iy are non-zero in two perpendicular directions
 2 unknowns, 2 equations
 Can measure actual motion

CS4243

Motion Tracking

31

Lucas-Kanade Method


Consider two consecutive image frames I and J:
I

J
x+d
d
x

x

moves from x = (x, y)T to x + d.
 d = (u, v) T
 Object

CS4243

Motion Tracking

32



So,



Or



Due to noise, there’s an error at position x:



Sum error over small window W at position x:

Similar to
template matching
CS4243

weight
Motion Tracking

33



If E is small, patterns in I and J match well.



So, find d that minimises E:
E / d = 0, compute d that minimises E.
 First, expand I(x – d) by Taylor’s series expansion:
 Set

 Omit

higher order terms:

 Write

in matrix form:

Intensity gradient
CS4243

Motion Tracking

34

 Now,

error E at position x is:

 Now,

differentiate E with respect to d (exercise):

 Setting

E / d = 0 gives

b

the only
unknown

Z

CS4243

Motion Tracking

35

 So,

2



CS4243

we get

unknowns, 2 equations. Can solve for d = (u, v).

What happen to aperture problem?
Did it disappear?
Motion Tracking

36

Lucas-Kanade + Tomasi


Lucas-Kanade algorithm is often used with
Tomasi’s feature
 Apply

Tomasi’s method to detect good features.
 Apply LK method to compute d for each pixel.
 Accept d only for good features.

CS4243

Motion Tracking

37

Example

Can you spot tracking errors?

CS4243

Motion Tracking

38

Constraints


Math of LK tracker assumes d is small.



In implementation, W is also small.



LK tracker is good only for small displacement.

CS4243

Motion Tracking

39

How to handle large displacement?


What if we scale down images?
 Displacements

CS4243

are smaller!

Motion Tracking

40

Image Pyramid
Usually, smoothen
image with Gaussian
filter before scaling
down.

downsample

scaled image

input image
CS4243

Motion Tracking

41

LK Tracking with Image Pyramid

CS4243

Motion Tracking

42



Construct image pyramids.



Apply LK tracker to low-resolution images.



Propagate results to higher-resolution images.



Apply LK tracker to higher-resolution images.

CS4243

Motion Tracking

43

Example

Tracking results are more accurate.

CS4243

Motion Tracking

44

Summary


Efficient algorithm, no explicit search.



Has aperture problem;
track good features only



LK tracker can’t track large displacement.



Use LK + image pyramid for large displacement.

CS4243

Motion Tracking

45

Software


OpenCV supports LK and LK with pyramid.



[Bir] offers LK with Tomasi’s features & pyramid.

CS4243

Motion Tracking

46

Appendix


Calculation of Ix, Iy, It
 Use

finite difference method
 Forward difference
Ix = I(x  1, y, t)  I(x, y, t)
Iy = I(x, y  1, t)  I(x, y, t)
It = I(x, y, t  1)  I(x, y, t)
 Backward difference
Ix = I(x, y, t)  I(x  1, y, t)
Iy = I(x, y, t)  I(x, y  1, t)
It = I(x, y, t)  I(x, y, t  1)

CS4243

Motion Tracking

47

Further Readings


Lucas-Kanade tracking with pyramid:
[BK08] Chapter 10.



Optical flow: [Sze10] Section 8.4.



Hierarchical motion estimation (with image
pyramid): [Sze10] Section 8.1.1.

CS4243

Motion Tracking

48

References


[Bir] S. Birchfield. KLT: An implementation of the Kanade-LucasTomasi feature tracker. http://vision.stanford.edu/~birch/klt/.



[BK08] Bradski and Kaehler. Learning OpenCV: Computer Vision
with the OpenCV Library. O’Reilly, 2008.



[LK81] B. D. Lucas and T. Kanade. An iterative image registration
technique with an application to stereo vision. In Proceedings of 7th
International Joint Conference on Artificial Intelligence, pages 674–
679, 1981.



[ST94] J. Shi and C. Tomasi. Good features to track. In Proceedings
of IEEE Conference on Computer Vision and Pattern Recognition,
pages 593–600, 1994.



[Sze10] R. Szeliski. Computer Vision: Algorithms and Applications.
Springer, 2010.

CS4243

Motion Tracking

49

References


CS4243

[TK91] C. Tomasi and T. Kanade. Detection and tracking of point
features. Technical Report CMU-CS-91-132, School of Computer
Science, Carnegie Mellon University, 1991.

Motion Tracking

50


Slide 18

Leow Wee Kheng
CS4243 Computer Vision and Pattern Recognition

Motion Tracking

CS4243

Motion Tracking

1

Changes are everywhere!

CS4243

Motion Tracking

2

Illumination change

CS4243

Motion Tracking

3

Shape change

CS4243

Motion Tracking

4

Object motion

CS4243

Motion Tracking

5

Camera motion

CS4243

Motion Tracking

6

Object & camera motion

CS4243

Motion Tracking

7

Everything changes!

CS4243

Motion Tracking

8

Motion analysis is tough in general!

We focus on object / camera motion.

CS4243

Motion Tracking

9

Change Detection


Detects any change in two video frames.



Straightforward method:
 Compute

difference between corresponding pixels:



: intensity / colour at (x, y) in frame t.
> threshold, has large difference.

 If

CS4243

Motion Tracking

10

Any difference?

No

CS4243

Motion Tracking

11

Any difference?

Yes,
illumination
change

CS4243

Motion Tracking

12

Any difference?

Yes,
position change

CS4243

Motion Tracking

13

Change Detection


Can detect
 Illumination

change
 Position change
 Illumination and position change


But, cannot distinguish between them.



Need to detect and measure position change.

CS4243

Motion Tracking

14

Motion Tracking


Two approaches
 Feature-based
 Intensity

CS4243

gradient-based

Motion Tracking

15

Feature-based Motion Tracking


Look for distinct features that change positions.

 Eagle’s

wing tips change positions.
 Tree tops don’t change positions.
CS4243

Motion Tracking

16

Basic Ideas
1.

Look for distinct features in current frame.

2.

For each feature


Search for matching feature within neighbourhood
in next frame.
 Difference in positions  displacement.
 Velocity = displacement / time difference.

CS4243

Motion Tracking

17

Basic Ideas
displacement

feature area

search area

CS4243

Motion Tracking

18

What feature to use?


Harris corner



Tomasi’s feature



Feature descriptors: SIFT, SURF, GLOH, etc.



Others

CS4243

Motion Tracking

19

Summary


Simple algorithm.



Can be slow if search area is large.



Can constrain search area with prior knowledge.

CS4243

Motion Tracking

20

Gradient-based Motion Tracking


Two basic assumptions
 Intensity

changes smoothly with position.
 Intensity of object doesn’t change over time.


Suppose an object is in motion.
 Change

position (dx, dy) over time dt.
 Then, from 2nd assumption:
 Apply

CS4243

Taylor’s series expansion:

Motion Tracking

21

 Omit

higher order terms and divide by dt

 Denote

 Then,

 u,

v are unknown.
 2 unknowns, 1 equation: can’t solve!

CS4243

Motion Tracking

22

Any difference in motion?

CS4243

Motion Tracking

23

Any difference in motion?

CS4243

Motion Tracking

24

Any difference in motion?

CS4243

Motion Tracking

25

Any difference in motion?

CS4243

Motion Tracking

26

Any difference in motion?

CS4243

Motion Tracking

27

Any difference in motion?

CS4243

Motion Tracking

28

Aperture Problem


Homogeneous region

 Ix

= Iy = It = 0.
 No change in local region.
 Cannot detect motion.

CS4243

Motion Tracking

29

Aperture Problem


Edge

 Ix

and Iy are zero along edge
 Cannot measure motion tangential to edge
 Ix and Iy are non-zero normal to edge
 Can measure motion normal to edge
 So, cannot measure actual motion

CS4243

Motion Tracking

30

Aperture Problem


Corner

 Ix

and Iy are non-zero in two perpendicular directions
 2 unknowns, 2 equations
 Can measure actual motion

CS4243

Motion Tracking

31

Lucas-Kanade Method


Consider two consecutive image frames I and J:
I

J
x+d
d
x

x

moves from x = (x, y)T to x + d.
 d = (u, v) T
 Object

CS4243

Motion Tracking

32



So,



Or



Due to noise, there’s an error at position x:



Sum error over small window W at position x:

Similar to
template matching
CS4243

weight
Motion Tracking

33



If E is small, patterns in I and J match well.



So, find d that minimises E:
E / d = 0, compute d that minimises E.
 First, expand I(x – d) by Taylor’s series expansion:
 Set

 Omit

higher order terms:

 Write

in matrix form:

Intensity gradient
CS4243

Motion Tracking

34

 Now,

error E at position x is:

 Now,

differentiate E with respect to d (exercise):

 Setting

E / d = 0 gives

b

the only
unknown

Z

CS4243

Motion Tracking

35

 So,

2



CS4243

we get

unknowns, 2 equations. Can solve for d = (u, v).

What happen to aperture problem?
Did it disappear?
Motion Tracking

36

Lucas-Kanade + Tomasi


Lucas-Kanade algorithm is often used with
Tomasi’s feature
 Apply

Tomasi’s method to detect good features.
 Apply LK method to compute d for each pixel.
 Accept d only for good features.

CS4243

Motion Tracking

37

Example

Can you spot tracking errors?

CS4243

Motion Tracking

38

Constraints


Math of LK tracker assumes d is small.



In implementation, W is also small.



LK tracker is good only for small displacement.

CS4243

Motion Tracking

39

How to handle large displacement?


What if we scale down images?
 Displacements

CS4243

are smaller!

Motion Tracking

40

Image Pyramid
Usually, smoothen
image with Gaussian
filter before scaling
down.

downsample

scaled image

input image
CS4243

Motion Tracking

41

LK Tracking with Image Pyramid

CS4243

Motion Tracking

42



Construct image pyramids.



Apply LK tracker to low-resolution images.



Propagate results to higher-resolution images.



Apply LK tracker to higher-resolution images.

CS4243

Motion Tracking

43

Example

Tracking results are more accurate.

CS4243

Motion Tracking

44

Summary


Efficient algorithm, no explicit search.



Has aperture problem;
track good features only



LK tracker can’t track large displacement.



Use LK + image pyramid for large displacement.

CS4243

Motion Tracking

45

Software


OpenCV supports LK and LK with pyramid.



[Bir] offers LK with Tomasi’s features & pyramid.

CS4243

Motion Tracking

46

Appendix


Calculation of Ix, Iy, It
 Use

finite difference method
 Forward difference
Ix = I(x  1, y, t)  I(x, y, t)
Iy = I(x, y  1, t)  I(x, y, t)
It = I(x, y, t  1)  I(x, y, t)
 Backward difference
Ix = I(x, y, t)  I(x  1, y, t)
Iy = I(x, y, t)  I(x, y  1, t)
It = I(x, y, t)  I(x, y, t  1)

CS4243

Motion Tracking

47

Further Readings


Lucas-Kanade tracking with pyramid:
[BK08] Chapter 10.



Optical flow: [Sze10] Section 8.4.



Hierarchical motion estimation (with image
pyramid): [Sze10] Section 8.1.1.

CS4243

Motion Tracking

48

References


[Bir] S. Birchfield. KLT: An implementation of the Kanade-LucasTomasi feature tracker. http://vision.stanford.edu/~birch/klt/.



[BK08] Bradski and Kaehler. Learning OpenCV: Computer Vision
with the OpenCV Library. O’Reilly, 2008.



[LK81] B. D. Lucas and T. Kanade. An iterative image registration
technique with an application to stereo vision. In Proceedings of 7th
International Joint Conference on Artificial Intelligence, pages 674–
679, 1981.



[ST94] J. Shi and C. Tomasi. Good features to track. In Proceedings
of IEEE Conference on Computer Vision and Pattern Recognition,
pages 593–600, 1994.



[Sze10] R. Szeliski. Computer Vision: Algorithms and Applications.
Springer, 2010.

CS4243

Motion Tracking

49

References


CS4243

[TK91] C. Tomasi and T. Kanade. Detection and tracking of point
features. Technical Report CMU-CS-91-132, School of Computer
Science, Carnegie Mellon University, 1991.

Motion Tracking

50


Slide 19

Leow Wee Kheng
CS4243 Computer Vision and Pattern Recognition

Motion Tracking

CS4243

Motion Tracking

1

Changes are everywhere!

CS4243

Motion Tracking

2

Illumination change

CS4243

Motion Tracking

3

Shape change

CS4243

Motion Tracking

4

Object motion

CS4243

Motion Tracking

5

Camera motion

CS4243

Motion Tracking

6

Object & camera motion

CS4243

Motion Tracking

7

Everything changes!

CS4243

Motion Tracking

8

Motion analysis is tough in general!

We focus on object / camera motion.

CS4243

Motion Tracking

9

Change Detection


Detects any change in two video frames.



Straightforward method:
 Compute

difference between corresponding pixels:



: intensity / colour at (x, y) in frame t.
> threshold, has large difference.

 If

CS4243

Motion Tracking

10

Any difference?

No

CS4243

Motion Tracking

11

Any difference?

Yes,
illumination
change

CS4243

Motion Tracking

12

Any difference?

Yes,
position change

CS4243

Motion Tracking

13

Change Detection


Can detect
 Illumination

change
 Position change
 Illumination and position change


But, cannot distinguish between them.



Need to detect and measure position change.

CS4243

Motion Tracking

14

Motion Tracking


Two approaches
 Feature-based
 Intensity

CS4243

gradient-based

Motion Tracking

15

Feature-based Motion Tracking


Look for distinct features that change positions.

 Eagle’s

wing tips change positions.
 Tree tops don’t change positions.
CS4243

Motion Tracking

16

Basic Ideas
1.

Look for distinct features in current frame.

2.

For each feature


Search for matching feature within neighbourhood
in next frame.
 Difference in positions  displacement.
 Velocity = displacement / time difference.

CS4243

Motion Tracking

17

Basic Ideas
displacement

feature area

search area

CS4243

Motion Tracking

18

What feature to use?


Harris corner



Tomasi’s feature



Feature descriptors: SIFT, SURF, GLOH, etc.



Others

CS4243

Motion Tracking

19

Summary


Simple algorithm.



Can be slow if search area is large.



Can constrain search area with prior knowledge.

CS4243

Motion Tracking

20

Gradient-based Motion Tracking


Two basic assumptions
 Intensity

changes smoothly with position.
 Intensity of object doesn’t change over time.


Suppose an object is in motion.
 Change

position (dx, dy) over time dt.
 Then, from 2nd assumption:
 Apply

CS4243

Taylor’s series expansion:

Motion Tracking

21

 Omit

higher order terms and divide by dt

 Denote

 Then,

 u,

v are unknown.
 2 unknowns, 1 equation: can’t solve!

CS4243

Motion Tracking

22

Any difference in motion?

CS4243

Motion Tracking

23

Any difference in motion?

CS4243

Motion Tracking

24

Any difference in motion?

CS4243

Motion Tracking

25

Any difference in motion?

CS4243

Motion Tracking

26

Any difference in motion?

CS4243

Motion Tracking

27

Any difference in motion?

CS4243

Motion Tracking

28

Aperture Problem


Homogeneous region

 Ix

= Iy = It = 0.
 No change in local region.
 Cannot detect motion.

CS4243

Motion Tracking

29

Aperture Problem


Edge

 Ix

and Iy are zero along edge
 Cannot measure motion tangential to edge
 Ix and Iy are non-zero normal to edge
 Can measure motion normal to edge
 So, cannot measure actual motion

CS4243

Motion Tracking

30

Aperture Problem


Corner

 Ix

and Iy are non-zero in two perpendicular directions
 2 unknowns, 2 equations
 Can measure actual motion

CS4243

Motion Tracking

31

Lucas-Kanade Method


Consider two consecutive image frames I and J:
I

J
x+d
d
x

x

moves from x = (x, y)T to x + d.
 d = (u, v) T
 Object

CS4243

Motion Tracking

32



So,



Or



Due to noise, there’s an error at position x:



Sum error over small window W at position x:

Similar to
template matching
CS4243

weight
Motion Tracking

33



If E is small, patterns in I and J match well.



So, find d that minimises E:
E / d = 0, compute d that minimises E.
 First, expand I(x – d) by Taylor’s series expansion:
 Set

 Omit

higher order terms:

 Write

in matrix form:

Intensity gradient
CS4243

Motion Tracking

34

 Now,

error E at position x is:

 Now,

differentiate E with respect to d (exercise):

 Setting

E / d = 0 gives

b

the only
unknown

Z

CS4243

Motion Tracking

35

 So,

2



CS4243

we get

unknowns, 2 equations. Can solve for d = (u, v).

What happen to aperture problem?
Did it disappear?
Motion Tracking

36

Lucas-Kanade + Tomasi


Lucas-Kanade algorithm is often used with
Tomasi’s feature
 Apply

Tomasi’s method to detect good features.
 Apply LK method to compute d for each pixel.
 Accept d only for good features.

CS4243

Motion Tracking

37

Example

Can you spot tracking errors?

CS4243

Motion Tracking

38

Constraints


Math of LK tracker assumes d is small.



In implementation, W is also small.



LK tracker is good only for small displacement.

CS4243

Motion Tracking

39

How to handle large displacement?


What if we scale down images?
 Displacements

CS4243

are smaller!

Motion Tracking

40

Image Pyramid
Usually, smoothen
image with Gaussian
filter before scaling
down.

downsample

scaled image

input image
CS4243

Motion Tracking

41

LK Tracking with Image Pyramid

CS4243

Motion Tracking

42



Construct image pyramids.



Apply LK tracker to low-resolution images.



Propagate results to higher-resolution images.



Apply LK tracker to higher-resolution images.

CS4243

Motion Tracking

43

Example

Tracking results are more accurate.

CS4243

Motion Tracking

44

Summary


Efficient algorithm, no explicit search.



Has aperture problem;
track good features only



LK tracker can’t track large displacement.



Use LK + image pyramid for large displacement.

CS4243

Motion Tracking

45

Software


OpenCV supports LK and LK with pyramid.



[Bir] offers LK with Tomasi’s features & pyramid.

CS4243

Motion Tracking

46

Appendix


Calculation of Ix, Iy, It
 Use

finite difference method
 Forward difference
Ix = I(x  1, y, t)  I(x, y, t)
Iy = I(x, y  1, t)  I(x, y, t)
It = I(x, y, t  1)  I(x, y, t)
 Backward difference
Ix = I(x, y, t)  I(x  1, y, t)
Iy = I(x, y, t)  I(x, y  1, t)
It = I(x, y, t)  I(x, y, t  1)

CS4243

Motion Tracking

47

Further Readings


Lucas-Kanade tracking with pyramid:
[BK08] Chapter 10.



Optical flow: [Sze10] Section 8.4.



Hierarchical motion estimation (with image
pyramid): [Sze10] Section 8.1.1.

CS4243

Motion Tracking

48

References


[Bir] S. Birchfield. KLT: An implementation of the Kanade-LucasTomasi feature tracker. http://vision.stanford.edu/~birch/klt/.



[BK08] Bradski and Kaehler. Learning OpenCV: Computer Vision
with the OpenCV Library. O’Reilly, 2008.



[LK81] B. D. Lucas and T. Kanade. An iterative image registration
technique with an application to stereo vision. In Proceedings of 7th
International Joint Conference on Artificial Intelligence, pages 674–
679, 1981.



[ST94] J. Shi and C. Tomasi. Good features to track. In Proceedings
of IEEE Conference on Computer Vision and Pattern Recognition,
pages 593–600, 1994.



[Sze10] R. Szeliski. Computer Vision: Algorithms and Applications.
Springer, 2010.

CS4243

Motion Tracking

49

References


CS4243

[TK91] C. Tomasi and T. Kanade. Detection and tracking of point
features. Technical Report CMU-CS-91-132, School of Computer
Science, Carnegie Mellon University, 1991.

Motion Tracking

50


Slide 20

Leow Wee Kheng
CS4243 Computer Vision and Pattern Recognition

Motion Tracking

CS4243

Motion Tracking

1

Changes are everywhere!

CS4243

Motion Tracking

2

Illumination change

CS4243

Motion Tracking

3

Shape change

CS4243

Motion Tracking

4

Object motion

CS4243

Motion Tracking

5

Camera motion

CS4243

Motion Tracking

6

Object & camera motion

CS4243

Motion Tracking

7

Everything changes!

CS4243

Motion Tracking

8

Motion analysis is tough in general!

We focus on object / camera motion.

CS4243

Motion Tracking

9

Change Detection


Detects any change in two video frames.



Straightforward method:
 Compute

difference between corresponding pixels:



: intensity / colour at (x, y) in frame t.
> threshold, has large difference.

 If

CS4243

Motion Tracking

10

Any difference?

No

CS4243

Motion Tracking

11

Any difference?

Yes,
illumination
change

CS4243

Motion Tracking

12

Any difference?

Yes,
position change

CS4243

Motion Tracking

13

Change Detection


Can detect
 Illumination

change
 Position change
 Illumination and position change


But, cannot distinguish between them.



Need to detect and measure position change.

CS4243

Motion Tracking

14

Motion Tracking


Two approaches
 Feature-based
 Intensity

CS4243

gradient-based

Motion Tracking

15

Feature-based Motion Tracking


Look for distinct features that change positions.

 Eagle’s

wing tips change positions.
 Tree tops don’t change positions.
CS4243

Motion Tracking

16

Basic Ideas
1.

Look for distinct features in current frame.

2.

For each feature


Search for matching feature within neighbourhood
in next frame.
 Difference in positions  displacement.
 Velocity = displacement / time difference.

CS4243

Motion Tracking

17

Basic Ideas
displacement

feature area

search area

CS4243

Motion Tracking

18

What feature to use?


Harris corner



Tomasi’s feature



Feature descriptors: SIFT, SURF, GLOH, etc.



Others

CS4243

Motion Tracking

19

Summary


Simple algorithm.



Can be slow if search area is large.



Can constrain search area with prior knowledge.

CS4243

Motion Tracking

20

Gradient-based Motion Tracking


Two basic assumptions
 Intensity

changes smoothly with position.
 Intensity of object doesn’t change over time.


Suppose an object is in motion.
 Change

position (dx, dy) over time dt.
 Then, from 2nd assumption:
 Apply

CS4243

Taylor’s series expansion:

Motion Tracking

21

 Omit

higher order terms and divide by dt

 Denote

 Then,

 u,

v are unknown.
 2 unknowns, 1 equation: can’t solve!

CS4243

Motion Tracking

22

Any difference in motion?

CS4243

Motion Tracking

23

Any difference in motion?

CS4243

Motion Tracking

24

Any difference in motion?

CS4243

Motion Tracking

25

Any difference in motion?

CS4243

Motion Tracking

26

Any difference in motion?

CS4243

Motion Tracking

27

Any difference in motion?

CS4243

Motion Tracking

28

Aperture Problem


Homogeneous region

 Ix

= Iy = It = 0.
 No change in local region.
 Cannot detect motion.

CS4243

Motion Tracking

29

Aperture Problem


Edge

 Ix

and Iy are zero along edge
 Cannot measure motion tangential to edge
 Ix and Iy are non-zero normal to edge
 Can measure motion normal to edge
 So, cannot measure actual motion

CS4243

Motion Tracking

30

Aperture Problem


Corner

 Ix

and Iy are non-zero in two perpendicular directions
 2 unknowns, 2 equations
 Can measure actual motion

CS4243

Motion Tracking

31

Lucas-Kanade Method


Consider two consecutive image frames I and J:
I

J
x+d
d
x

x

moves from x = (x, y)T to x + d.
 d = (u, v) T
 Object

CS4243

Motion Tracking

32



So,



Or



Due to noise, there’s an error at position x:



Sum error over small window W at position x:

Similar to
template matching
CS4243

weight
Motion Tracking

33



If E is small, patterns in I and J match well.



So, find d that minimises E:
E / d = 0, compute d that minimises E.
 First, expand I(x – d) by Taylor’s series expansion:
 Set

 Omit

higher order terms:

 Write

in matrix form:

Intensity gradient
CS4243

Motion Tracking

34

 Now,

error E at position x is:

 Now,

differentiate E with respect to d (exercise):

 Setting

E / d = 0 gives

b

the only
unknown

Z

CS4243

Motion Tracking

35

 So,

2



CS4243

we get

unknowns, 2 equations. Can solve for d = (u, v).

What happen to aperture problem?
Did it disappear?
Motion Tracking

36

Lucas-Kanade + Tomasi


Lucas-Kanade algorithm is often used with
Tomasi’s feature
 Apply

Tomasi’s method to detect good features.
 Apply LK method to compute d for each pixel.
 Accept d only for good features.

CS4243

Motion Tracking

37

Example

Can you spot tracking errors?

CS4243

Motion Tracking

38

Constraints


Math of LK tracker assumes d is small.



In implementation, W is also small.



LK tracker is good only for small displacement.

CS4243

Motion Tracking

39

How to handle large displacement?


What if we scale down images?
 Displacements

CS4243

are smaller!

Motion Tracking

40

Image Pyramid
Usually, smoothen
image with Gaussian
filter before scaling
down.

downsample

scaled image

input image
CS4243

Motion Tracking

41

LK Tracking with Image Pyramid

CS4243

Motion Tracking

42



Construct image pyramids.



Apply LK tracker to low-resolution images.



Propagate results to higher-resolution images.



Apply LK tracker to higher-resolution images.

CS4243

Motion Tracking

43

Example

Tracking results are more accurate.

CS4243

Motion Tracking

44

Summary


Efficient algorithm, no explicit search.



Has aperture problem;
track good features only



LK tracker can’t track large displacement.



Use LK + image pyramid for large displacement.

CS4243

Motion Tracking

45

Software


OpenCV supports LK and LK with pyramid.



[Bir] offers LK with Tomasi’s features & pyramid.

CS4243

Motion Tracking

46

Appendix


Calculation of Ix, Iy, It
 Use

finite difference method
 Forward difference
Ix = I(x  1, y, t)  I(x, y, t)
Iy = I(x, y  1, t)  I(x, y, t)
It = I(x, y, t  1)  I(x, y, t)
 Backward difference
Ix = I(x, y, t)  I(x  1, y, t)
Iy = I(x, y, t)  I(x, y  1, t)
It = I(x, y, t)  I(x, y, t  1)

CS4243

Motion Tracking

47

Further Readings


Lucas-Kanade tracking with pyramid:
[BK08] Chapter 10.



Optical flow: [Sze10] Section 8.4.



Hierarchical motion estimation (with image
pyramid): [Sze10] Section 8.1.1.

CS4243

Motion Tracking

48

References


[Bir] S. Birchfield. KLT: An implementation of the Kanade-LucasTomasi feature tracker. http://vision.stanford.edu/~birch/klt/.



[BK08] Bradski and Kaehler. Learning OpenCV: Computer Vision
with the OpenCV Library. O’Reilly, 2008.



[LK81] B. D. Lucas and T. Kanade. An iterative image registration
technique with an application to stereo vision. In Proceedings of 7th
International Joint Conference on Artificial Intelligence, pages 674–
679, 1981.



[ST94] J. Shi and C. Tomasi. Good features to track. In Proceedings
of IEEE Conference on Computer Vision and Pattern Recognition,
pages 593–600, 1994.



[Sze10] R. Szeliski. Computer Vision: Algorithms and Applications.
Springer, 2010.

CS4243

Motion Tracking

49

References


CS4243

[TK91] C. Tomasi and T. Kanade. Detection and tracking of point
features. Technical Report CMU-CS-91-132, School of Computer
Science, Carnegie Mellon University, 1991.

Motion Tracking

50


Slide 21

Leow Wee Kheng
CS4243 Computer Vision and Pattern Recognition

Motion Tracking

CS4243

Motion Tracking

1

Changes are everywhere!

CS4243

Motion Tracking

2

Illumination change

CS4243

Motion Tracking

3

Shape change

CS4243

Motion Tracking

4

Object motion

CS4243

Motion Tracking

5

Camera motion

CS4243

Motion Tracking

6

Object & camera motion

CS4243

Motion Tracking

7

Everything changes!

CS4243

Motion Tracking

8

Motion analysis is tough in general!

We focus on object / camera motion.

CS4243

Motion Tracking

9

Change Detection


Detects any change in two video frames.



Straightforward method:
 Compute

difference between corresponding pixels:



: intensity / colour at (x, y) in frame t.
> threshold, has large difference.

 If

CS4243

Motion Tracking

10

Any difference?

No

CS4243

Motion Tracking

11

Any difference?

Yes,
illumination
change

CS4243

Motion Tracking

12

Any difference?

Yes,
position change

CS4243

Motion Tracking

13

Change Detection


Can detect
 Illumination

change
 Position change
 Illumination and position change


But, cannot distinguish between them.



Need to detect and measure position change.

CS4243

Motion Tracking

14

Motion Tracking


Two approaches
 Feature-based
 Intensity

CS4243

gradient-based

Motion Tracking

15

Feature-based Motion Tracking


Look for distinct features that change positions.

 Eagle’s

wing tips change positions.
 Tree tops don’t change positions.
CS4243

Motion Tracking

16

Basic Ideas
1.

Look for distinct features in current frame.

2.

For each feature


Search for matching feature within neighbourhood
in next frame.
 Difference in positions  displacement.
 Velocity = displacement / time difference.

CS4243

Motion Tracking

17

Basic Ideas
displacement

feature area

search area

CS4243

Motion Tracking

18

What feature to use?


Harris corner



Tomasi’s feature



Feature descriptors: SIFT, SURF, GLOH, etc.



Others

CS4243

Motion Tracking

19

Summary


Simple algorithm.



Can be slow if search area is large.



Can constrain search area with prior knowledge.

CS4243

Motion Tracking

20

Gradient-based Motion Tracking


Two basic assumptions
 Intensity

changes smoothly with position.
 Intensity of object doesn’t change over time.


Suppose an object is in motion.
 Change

position (dx, dy) over time dt.
 Then, from 2nd assumption:
 Apply

CS4243

Taylor’s series expansion:

Motion Tracking

21

 Omit

higher order terms and divide by dt

 Denote

 Then,

 u,

v are unknown.
 2 unknowns, 1 equation: can’t solve!

CS4243

Motion Tracking

22

Any difference in motion?

CS4243

Motion Tracking

23

Any difference in motion?

CS4243

Motion Tracking

24

Any difference in motion?

CS4243

Motion Tracking

25

Any difference in motion?

CS4243

Motion Tracking

26

Any difference in motion?

CS4243

Motion Tracking

27

Any difference in motion?

CS4243

Motion Tracking

28

Aperture Problem


Homogeneous region

 Ix

= Iy = It = 0.
 No change in local region.
 Cannot detect motion.

CS4243

Motion Tracking

29

Aperture Problem


Edge

 Ix

and Iy are zero along edge
 Cannot measure motion tangential to edge
 Ix and Iy are non-zero normal to edge
 Can measure motion normal to edge
 So, cannot measure actual motion

CS4243

Motion Tracking

30

Aperture Problem


Corner

 Ix

and Iy are non-zero in two perpendicular directions
 2 unknowns, 2 equations
 Can measure actual motion

CS4243

Motion Tracking

31

Lucas-Kanade Method


Consider two consecutive image frames I and J:
I

J
x+d
d
x

x

moves from x = (x, y)T to x + d.
 d = (u, v) T
 Object

CS4243

Motion Tracking

32



So,



Or



Due to noise, there’s an error at position x:



Sum error over small window W at position x:

Similar to
template matching
CS4243

weight
Motion Tracking

33



If E is small, patterns in I and J match well.



So, find d that minimises E:
E / d = 0, compute d that minimises E.
 First, expand I(x – d) by Taylor’s series expansion:
 Set

 Omit

higher order terms:

 Write

in matrix form:

Intensity gradient
CS4243

Motion Tracking

34

 Now,

error E at position x is:

 Now,

differentiate E with respect to d (exercise):

 Setting

E / d = 0 gives

b

the only
unknown

Z

CS4243

Motion Tracking

35

 So,

2



CS4243

we get

unknowns, 2 equations. Can solve for d = (u, v).

What happen to aperture problem?
Did it disappear?
Motion Tracking

36

Lucas-Kanade + Tomasi


Lucas-Kanade algorithm is often used with
Tomasi’s feature
 Apply

Tomasi’s method to detect good features.
 Apply LK method to compute d for each pixel.
 Accept d only for good features.

CS4243

Motion Tracking

37

Example

Can you spot tracking errors?

CS4243

Motion Tracking

38

Constraints


Math of LK tracker assumes d is small.



In implementation, W is also small.



LK tracker is good only for small displacement.

CS4243

Motion Tracking

39

How to handle large displacement?


What if we scale down images?
 Displacements

CS4243

are smaller!

Motion Tracking

40

Image Pyramid
Usually, smoothen
image with Gaussian
filter before scaling
down.

downsample

scaled image

input image
CS4243

Motion Tracking

41

LK Tracking with Image Pyramid

CS4243

Motion Tracking

42



Construct image pyramids.



Apply LK tracker to low-resolution images.



Propagate results to higher-resolution images.



Apply LK tracker to higher-resolution images.

CS4243

Motion Tracking

43

Example

Tracking results are more accurate.

CS4243

Motion Tracking

44

Summary


Efficient algorithm, no explicit search.



Has aperture problem;
track good features only



LK tracker can’t track large displacement.



Use LK + image pyramid for large displacement.

CS4243

Motion Tracking

45

Software


OpenCV supports LK and LK with pyramid.



[Bir] offers LK with Tomasi’s features & pyramid.

CS4243

Motion Tracking

46

Appendix


Calculation of Ix, Iy, It
 Use

finite difference method
 Forward difference
Ix = I(x  1, y, t)  I(x, y, t)
Iy = I(x, y  1, t)  I(x, y, t)
It = I(x, y, t  1)  I(x, y, t)
 Backward difference
Ix = I(x, y, t)  I(x  1, y, t)
Iy = I(x, y, t)  I(x, y  1, t)
It = I(x, y, t)  I(x, y, t  1)

CS4243

Motion Tracking

47

Further Readings


Lucas-Kanade tracking with pyramid:
[BK08] Chapter 10.



Optical flow: [Sze10] Section 8.4.



Hierarchical motion estimation (with image
pyramid): [Sze10] Section 8.1.1.

CS4243

Motion Tracking

48

References


[Bir] S. Birchfield. KLT: An implementation of the Kanade-LucasTomasi feature tracker. http://vision.stanford.edu/~birch/klt/.



[BK08] Bradski and Kaehler. Learning OpenCV: Computer Vision
with the OpenCV Library. O’Reilly, 2008.



[LK81] B. D. Lucas and T. Kanade. An iterative image registration
technique with an application to stereo vision. In Proceedings of 7th
International Joint Conference on Artificial Intelligence, pages 674–
679, 1981.



[ST94] J. Shi and C. Tomasi. Good features to track. In Proceedings
of IEEE Conference on Computer Vision and Pattern Recognition,
pages 593–600, 1994.



[Sze10] R. Szeliski. Computer Vision: Algorithms and Applications.
Springer, 2010.

CS4243

Motion Tracking

49

References


CS4243

[TK91] C. Tomasi and T. Kanade. Detection and tracking of point
features. Technical Report CMU-CS-91-132, School of Computer
Science, Carnegie Mellon University, 1991.

Motion Tracking

50


Slide 22

Leow Wee Kheng
CS4243 Computer Vision and Pattern Recognition

Motion Tracking

CS4243

Motion Tracking

1

Changes are everywhere!

CS4243

Motion Tracking

2

Illumination change

CS4243

Motion Tracking

3

Shape change

CS4243

Motion Tracking

4

Object motion

CS4243

Motion Tracking

5

Camera motion

CS4243

Motion Tracking

6

Object & camera motion

CS4243

Motion Tracking

7

Everything changes!

CS4243

Motion Tracking

8

Motion analysis is tough in general!

We focus on object / camera motion.

CS4243

Motion Tracking

9

Change Detection


Detects any change in two video frames.



Straightforward method:
 Compute

difference between corresponding pixels:



: intensity / colour at (x, y) in frame t.
> threshold, has large difference.

 If

CS4243

Motion Tracking

10

Any difference?

No

CS4243

Motion Tracking

11

Any difference?

Yes,
illumination
change

CS4243

Motion Tracking

12

Any difference?

Yes,
position change

CS4243

Motion Tracking

13

Change Detection


Can detect
 Illumination

change
 Position change
 Illumination and position change


But, cannot distinguish between them.



Need to detect and measure position change.

CS4243

Motion Tracking

14

Motion Tracking


Two approaches
 Feature-based
 Intensity

CS4243

gradient-based

Motion Tracking

15

Feature-based Motion Tracking


Look for distinct features that change positions.

 Eagle’s

wing tips change positions.
 Tree tops don’t change positions.
CS4243

Motion Tracking

16

Basic Ideas
1.

Look for distinct features in current frame.

2.

For each feature


Search for matching feature within neighbourhood
in next frame.
 Difference in positions  displacement.
 Velocity = displacement / time difference.

CS4243

Motion Tracking

17

Basic Ideas
displacement

feature area

search area

CS4243

Motion Tracking

18

What feature to use?


Harris corner



Tomasi’s feature



Feature descriptors: SIFT, SURF, GLOH, etc.



Others

CS4243

Motion Tracking

19

Summary


Simple algorithm.



Can be slow if search area is large.



Can constrain search area with prior knowledge.

CS4243

Motion Tracking

20

Gradient-based Motion Tracking


Two basic assumptions
 Intensity

changes smoothly with position.
 Intensity of object doesn’t change over time.


Suppose an object is in motion.
 Change

position (dx, dy) over time dt.
 Then, from 2nd assumption:
 Apply

CS4243

Taylor’s series expansion:

Motion Tracking

21

 Omit

higher order terms and divide by dt

 Denote

 Then,

 u,

v are unknown.
 2 unknowns, 1 equation: can’t solve!

CS4243

Motion Tracking

22

Any difference in motion?

CS4243

Motion Tracking

23

Any difference in motion?

CS4243

Motion Tracking

24

Any difference in motion?

CS4243

Motion Tracking

25

Any difference in motion?

CS4243

Motion Tracking

26

Any difference in motion?

CS4243

Motion Tracking

27

Any difference in motion?

CS4243

Motion Tracking

28

Aperture Problem


Homogeneous region

 Ix

= Iy = It = 0.
 No change in local region.
 Cannot detect motion.

CS4243

Motion Tracking

29

Aperture Problem


Edge

 Ix

and Iy are zero along edge
 Cannot measure motion tangential to edge
 Ix and Iy are non-zero normal to edge
 Can measure motion normal to edge
 So, cannot measure actual motion

CS4243

Motion Tracking

30

Aperture Problem


Corner

 Ix

and Iy are non-zero in two perpendicular directions
 2 unknowns, 2 equations
 Can measure actual motion

CS4243

Motion Tracking

31

Lucas-Kanade Method


Consider two consecutive image frames I and J:
I

J
x+d
d
x

x

moves from x = (x, y)T to x + d.
 d = (u, v) T
 Object

CS4243

Motion Tracking

32



So,



Or



Due to noise, there’s an error at position x:



Sum error over small window W at position x:

Similar to
template matching
CS4243

weight
Motion Tracking

33



If E is small, patterns in I and J match well.



So, find d that minimises E:
E / d = 0, compute d that minimises E.
 First, expand I(x – d) by Taylor’s series expansion:
 Set

 Omit

higher order terms:

 Write

in matrix form:

Intensity gradient
CS4243

Motion Tracking

34

 Now,

error E at position x is:

 Now,

differentiate E with respect to d (exercise):

 Setting

E / d = 0 gives

b

the only
unknown

Z

CS4243

Motion Tracking

35

 So,

2



CS4243

we get

unknowns, 2 equations. Can solve for d = (u, v).

What happen to aperture problem?
Did it disappear?
Motion Tracking

36

Lucas-Kanade + Tomasi


Lucas-Kanade algorithm is often used with
Tomasi’s feature
 Apply

Tomasi’s method to detect good features.
 Apply LK method to compute d for each pixel.
 Accept d only for good features.

CS4243

Motion Tracking

37

Example

Can you spot tracking errors?

CS4243

Motion Tracking

38

Constraints


Math of LK tracker assumes d is small.



In implementation, W is also small.



LK tracker is good only for small displacement.

CS4243

Motion Tracking

39

How to handle large displacement?


What if we scale down images?
 Displacements

CS4243

are smaller!

Motion Tracking

40

Image Pyramid
Usually, smoothen
image with Gaussian
filter before scaling
down.

downsample

scaled image

input image
CS4243

Motion Tracking

41

LK Tracking with Image Pyramid

CS4243

Motion Tracking

42



Construct image pyramids.



Apply LK tracker to low-resolution images.



Propagate results to higher-resolution images.



Apply LK tracker to higher-resolution images.

CS4243

Motion Tracking

43

Example

Tracking results are more accurate.

CS4243

Motion Tracking

44

Summary


Efficient algorithm, no explicit search.



Has aperture problem;
track good features only



LK tracker can’t track large displacement.



Use LK + image pyramid for large displacement.

CS4243

Motion Tracking

45

Software


OpenCV supports LK and LK with pyramid.



[Bir] offers LK with Tomasi’s features & pyramid.

CS4243

Motion Tracking

46

Appendix


Calculation of Ix, Iy, It
 Use

finite difference method
 Forward difference
Ix = I(x  1, y, t)  I(x, y, t)
Iy = I(x, y  1, t)  I(x, y, t)
It = I(x, y, t  1)  I(x, y, t)
 Backward difference
Ix = I(x, y, t)  I(x  1, y, t)
Iy = I(x, y, t)  I(x, y  1, t)
It = I(x, y, t)  I(x, y, t  1)

CS4243

Motion Tracking

47

Further Readings


Lucas-Kanade tracking with pyramid:
[BK08] Chapter 10.



Optical flow: [Sze10] Section 8.4.



Hierarchical motion estimation (with image
pyramid): [Sze10] Section 8.1.1.

CS4243

Motion Tracking

48

References


[Bir] S. Birchfield. KLT: An implementation of the Kanade-LucasTomasi feature tracker. http://vision.stanford.edu/~birch/klt/.



[BK08] Bradski and Kaehler. Learning OpenCV: Computer Vision
with the OpenCV Library. O’Reilly, 2008.



[LK81] B. D. Lucas and T. Kanade. An iterative image registration
technique with an application to stereo vision. In Proceedings of 7th
International Joint Conference on Artificial Intelligence, pages 674–
679, 1981.



[ST94] J. Shi and C. Tomasi. Good features to track. In Proceedings
of IEEE Conference on Computer Vision and Pattern Recognition,
pages 593–600, 1994.



[Sze10] R. Szeliski. Computer Vision: Algorithms and Applications.
Springer, 2010.

CS4243

Motion Tracking

49

References


CS4243

[TK91] C. Tomasi and T. Kanade. Detection and tracking of point
features. Technical Report CMU-CS-91-132, School of Computer
Science, Carnegie Mellon University, 1991.

Motion Tracking

50


Slide 23

Leow Wee Kheng
CS4243 Computer Vision and Pattern Recognition

Motion Tracking

CS4243

Motion Tracking

1

Changes are everywhere!

CS4243

Motion Tracking

2

Illumination change

CS4243

Motion Tracking

3

Shape change

CS4243

Motion Tracking

4

Object motion

CS4243

Motion Tracking

5

Camera motion

CS4243

Motion Tracking

6

Object & camera motion

CS4243

Motion Tracking

7

Everything changes!

CS4243

Motion Tracking

8

Motion analysis is tough in general!

We focus on object / camera motion.

CS4243

Motion Tracking

9

Change Detection


Detects any change in two video frames.



Straightforward method:
 Compute

difference between corresponding pixels:



: intensity / colour at (x, y) in frame t.
> threshold, has large difference.

 If

CS4243

Motion Tracking

10

Any difference?

No

CS4243

Motion Tracking

11

Any difference?

Yes,
illumination
change

CS4243

Motion Tracking

12

Any difference?

Yes,
position change

CS4243

Motion Tracking

13

Change Detection


Can detect
 Illumination

change
 Position change
 Illumination and position change


But, cannot distinguish between them.



Need to detect and measure position change.

CS4243

Motion Tracking

14

Motion Tracking


Two approaches
 Feature-based
 Intensity

CS4243

gradient-based

Motion Tracking

15

Feature-based Motion Tracking


Look for distinct features that change positions.

 Eagle’s

wing tips change positions.
 Tree tops don’t change positions.
CS4243

Motion Tracking

16

Basic Ideas
1.

Look for distinct features in current frame.

2.

For each feature


Search for matching feature within neighbourhood
in next frame.
 Difference in positions  displacement.
 Velocity = displacement / time difference.

CS4243

Motion Tracking

17

Basic Ideas
displacement

feature area

search area

CS4243

Motion Tracking

18

What feature to use?


Harris corner



Tomasi’s feature



Feature descriptors: SIFT, SURF, GLOH, etc.



Others

CS4243

Motion Tracking

19

Summary


Simple algorithm.



Can be slow if search area is large.



Can constrain search area with prior knowledge.

CS4243

Motion Tracking

20

Gradient-based Motion Tracking


Two basic assumptions
 Intensity

changes smoothly with position.
 Intensity of object doesn’t change over time.


Suppose an object is in motion.
 Change

position (dx, dy) over time dt.
 Then, from 2nd assumption:
 Apply

CS4243

Taylor’s series expansion:

Motion Tracking

21

 Omit

higher order terms and divide by dt

 Denote

 Then,

 u,

v are unknown.
 2 unknowns, 1 equation: can’t solve!

CS4243

Motion Tracking

22

Any difference in motion?

CS4243

Motion Tracking

23

Any difference in motion?

CS4243

Motion Tracking

24

Any difference in motion?

CS4243

Motion Tracking

25

Any difference in motion?

CS4243

Motion Tracking

26

Any difference in motion?

CS4243

Motion Tracking

27

Any difference in motion?

CS4243

Motion Tracking

28

Aperture Problem


Homogeneous region

 Ix

= Iy = It = 0.
 No change in local region.
 Cannot detect motion.

CS4243

Motion Tracking

29

Aperture Problem


Edge

 Ix

and Iy are zero along edge
 Cannot measure motion tangential to edge
 Ix and Iy are non-zero normal to edge
 Can measure motion normal to edge
 So, cannot measure actual motion

CS4243

Motion Tracking

30

Aperture Problem


Corner

 Ix

and Iy are non-zero in two perpendicular directions
 2 unknowns, 2 equations
 Can measure actual motion

CS4243

Motion Tracking

31

Lucas-Kanade Method


Consider two consecutive image frames I and J:
I

J
x+d
d
x

x

moves from x = (x, y)T to x + d.
 d = (u, v) T
 Object

CS4243

Motion Tracking

32



So,



Or



Due to noise, there’s an error at position x:



Sum error over small window W at position x:

Similar to
template matching
CS4243

weight
Motion Tracking

33



If E is small, patterns in I and J match well.



So, find d that minimises E:
E / d = 0, compute d that minimises E.
 First, expand I(x – d) by Taylor’s series expansion:
 Set

 Omit

higher order terms:

 Write

in matrix form:

Intensity gradient
CS4243

Motion Tracking

34

 Now,

error E at position x is:

 Now,

differentiate E with respect to d (exercise):

 Setting

E / d = 0 gives

b

the only
unknown

Z

CS4243

Motion Tracking

35

 So,

2



CS4243

we get

unknowns, 2 equations. Can solve for d = (u, v).

What happen to aperture problem?
Did it disappear?
Motion Tracking

36

Lucas-Kanade + Tomasi


Lucas-Kanade algorithm is often used with
Tomasi’s feature
 Apply

Tomasi’s method to detect good features.
 Apply LK method to compute d for each pixel.
 Accept d only for good features.

CS4243

Motion Tracking

37

Example

Can you spot tracking errors?

CS4243

Motion Tracking

38

Constraints


Math of LK tracker assumes d is small.



In implementation, W is also small.



LK tracker is good only for small displacement.

CS4243

Motion Tracking

39

How to handle large displacement?


What if we scale down images?
 Displacements

CS4243

are smaller!

Motion Tracking

40

Image Pyramid
Usually, smoothen
image with Gaussian
filter before scaling
down.

downsample

scaled image

input image
CS4243

Motion Tracking

41

LK Tracking with Image Pyramid

CS4243

Motion Tracking

42



Construct image pyramids.



Apply LK tracker to low-resolution images.



Propagate results to higher-resolution images.



Apply LK tracker to higher-resolution images.

CS4243

Motion Tracking

43

Example

Tracking results are more accurate.

CS4243

Motion Tracking

44

Summary


Efficient algorithm, no explicit search.



Has aperture problem;
track good features only



LK tracker can’t track large displacement.



Use LK + image pyramid for large displacement.

CS4243

Motion Tracking

45

Software


OpenCV supports LK and LK with pyramid.



[Bir] offers LK with Tomasi’s features & pyramid.

CS4243

Motion Tracking

46

Appendix


Calculation of Ix, Iy, It
 Use

finite difference method
 Forward difference
Ix = I(x  1, y, t)  I(x, y, t)
Iy = I(x, y  1, t)  I(x, y, t)
It = I(x, y, t  1)  I(x, y, t)
 Backward difference
Ix = I(x, y, t)  I(x  1, y, t)
Iy = I(x, y, t)  I(x, y  1, t)
It = I(x, y, t)  I(x, y, t  1)

CS4243

Motion Tracking

47

Further Readings


Lucas-Kanade tracking with pyramid:
[BK08] Chapter 10.



Optical flow: [Sze10] Section 8.4.



Hierarchical motion estimation (with image
pyramid): [Sze10] Section 8.1.1.

CS4243

Motion Tracking

48

References


[Bir] S. Birchfield. KLT: An implementation of the Kanade-LucasTomasi feature tracker. http://vision.stanford.edu/~birch/klt/.



[BK08] Bradski and Kaehler. Learning OpenCV: Computer Vision
with the OpenCV Library. O’Reilly, 2008.



[LK81] B. D. Lucas and T. Kanade. An iterative image registration
technique with an application to stereo vision. In Proceedings of 7th
International Joint Conference on Artificial Intelligence, pages 674–
679, 1981.



[ST94] J. Shi and C. Tomasi. Good features to track. In Proceedings
of IEEE Conference on Computer Vision and Pattern Recognition,
pages 593–600, 1994.



[Sze10] R. Szeliski. Computer Vision: Algorithms and Applications.
Springer, 2010.

CS4243

Motion Tracking

49

References


CS4243

[TK91] C. Tomasi and T. Kanade. Detection and tracking of point
features. Technical Report CMU-CS-91-132, School of Computer
Science, Carnegie Mellon University, 1991.

Motion Tracking

50


Slide 24

Leow Wee Kheng
CS4243 Computer Vision and Pattern Recognition

Motion Tracking

CS4243

Motion Tracking

1

Changes are everywhere!

CS4243

Motion Tracking

2

Illumination change

CS4243

Motion Tracking

3

Shape change

CS4243

Motion Tracking

4

Object motion

CS4243

Motion Tracking

5

Camera motion

CS4243

Motion Tracking

6

Object & camera motion

CS4243

Motion Tracking

7

Everything changes!

CS4243

Motion Tracking

8

Motion analysis is tough in general!

We focus on object / camera motion.

CS4243

Motion Tracking

9

Change Detection


Detects any change in two video frames.



Straightforward method:
 Compute

difference between corresponding pixels:



: intensity / colour at (x, y) in frame t.
> threshold, has large difference.

 If

CS4243

Motion Tracking

10

Any difference?

No

CS4243

Motion Tracking

11

Any difference?

Yes,
illumination
change

CS4243

Motion Tracking

12

Any difference?

Yes,
position change

CS4243

Motion Tracking

13

Change Detection


Can detect
 Illumination

change
 Position change
 Illumination and position change


But, cannot distinguish between them.



Need to detect and measure position change.

CS4243

Motion Tracking

14

Motion Tracking


Two approaches
 Feature-based
 Intensity

CS4243

gradient-based

Motion Tracking

15

Feature-based Motion Tracking


Look for distinct features that change positions.

 Eagle’s

wing tips change positions.
 Tree tops don’t change positions.
CS4243

Motion Tracking

16

Basic Ideas
1.

Look for distinct features in current frame.

2.

For each feature


Search for matching feature within neighbourhood
in next frame.
 Difference in positions  displacement.
 Velocity = displacement / time difference.

CS4243

Motion Tracking

17

Basic Ideas
displacement

feature area

search area

CS4243

Motion Tracking

18

What feature to use?


Harris corner



Tomasi’s feature



Feature descriptors: SIFT, SURF, GLOH, etc.



Others

CS4243

Motion Tracking

19

Summary


Simple algorithm.



Can be slow if search area is large.



Can constrain search area with prior knowledge.

CS4243

Motion Tracking

20

Gradient-based Motion Tracking


Two basic assumptions
 Intensity

changes smoothly with position.
 Intensity of object doesn’t change over time.


Suppose an object is in motion.
 Change

position (dx, dy) over time dt.
 Then, from 2nd assumption:
 Apply

CS4243

Taylor’s series expansion:

Motion Tracking

21

 Omit

higher order terms and divide by dt

 Denote

 Then,

 u,

v are unknown.
 2 unknowns, 1 equation: can’t solve!

CS4243

Motion Tracking

22

Any difference in motion?

CS4243

Motion Tracking

23

Any difference in motion?

CS4243

Motion Tracking

24

Any difference in motion?

CS4243

Motion Tracking

25

Any difference in motion?

CS4243

Motion Tracking

26

Any difference in motion?

CS4243

Motion Tracking

27

Any difference in motion?

CS4243

Motion Tracking

28

Aperture Problem


Homogeneous region

 Ix

= Iy = It = 0.
 No change in local region.
 Cannot detect motion.

CS4243

Motion Tracking

29

Aperture Problem


Edge

 Ix

and Iy are zero along edge
 Cannot measure motion tangential to edge
 Ix and Iy are non-zero normal to edge
 Can measure motion normal to edge
 So, cannot measure actual motion

CS4243

Motion Tracking

30

Aperture Problem


Corner

 Ix

and Iy are non-zero in two perpendicular directions
 2 unknowns, 2 equations
 Can measure actual motion

CS4243

Motion Tracking

31

Lucas-Kanade Method


Consider two consecutive image frames I and J:
I

J
x+d
d
x

x

moves from x = (x, y)T to x + d.
 d = (u, v) T
 Object

CS4243

Motion Tracking

32



So,



Or



Due to noise, there’s an error at position x:



Sum error over small window W at position x:

Similar to
template matching
CS4243

weight
Motion Tracking

33



If E is small, patterns in I and J match well.



So, find d that minimises E:
E / d = 0, compute d that minimises E.
 First, expand I(x – d) by Taylor’s series expansion:
 Set

 Omit

higher order terms:

 Write

in matrix form:

Intensity gradient
CS4243

Motion Tracking

34

 Now,

error E at position x is:

 Now,

differentiate E with respect to d (exercise):

 Setting

E / d = 0 gives

b

the only
unknown

Z

CS4243

Motion Tracking

35

 So,

2



CS4243

we get

unknowns, 2 equations. Can solve for d = (u, v).

What happen to aperture problem?
Did it disappear?
Motion Tracking

36

Lucas-Kanade + Tomasi


Lucas-Kanade algorithm is often used with
Tomasi’s feature
 Apply

Tomasi’s method to detect good features.
 Apply LK method to compute d for each pixel.
 Accept d only for good features.

CS4243

Motion Tracking

37

Example

Can you spot tracking errors?

CS4243

Motion Tracking

38

Constraints


Math of LK tracker assumes d is small.



In implementation, W is also small.



LK tracker is good only for small displacement.

CS4243

Motion Tracking

39

How to handle large displacement?


What if we scale down images?
 Displacements

CS4243

are smaller!

Motion Tracking

40

Image Pyramid
Usually, smoothen
image with Gaussian
filter before scaling
down.

downsample

scaled image

input image
CS4243

Motion Tracking

41

LK Tracking with Image Pyramid

CS4243

Motion Tracking

42



Construct image pyramids.



Apply LK tracker to low-resolution images.



Propagate results to higher-resolution images.



Apply LK tracker to higher-resolution images.

CS4243

Motion Tracking

43

Example

Tracking results are more accurate.

CS4243

Motion Tracking

44

Summary


Efficient algorithm, no explicit search.



Has aperture problem;
track good features only



LK tracker can’t track large displacement.



Use LK + image pyramid for large displacement.

CS4243

Motion Tracking

45

Software


OpenCV supports LK and LK with pyramid.



[Bir] offers LK with Tomasi’s features & pyramid.

CS4243

Motion Tracking

46

Appendix


Calculation of Ix, Iy, It
 Use

finite difference method
 Forward difference
Ix = I(x  1, y, t)  I(x, y, t)
Iy = I(x, y  1, t)  I(x, y, t)
It = I(x, y, t  1)  I(x, y, t)
 Backward difference
Ix = I(x, y, t)  I(x  1, y, t)
Iy = I(x, y, t)  I(x, y  1, t)
It = I(x, y, t)  I(x, y, t  1)

CS4243

Motion Tracking

47

Further Readings


Lucas-Kanade tracking with pyramid:
[BK08] Chapter 10.



Optical flow: [Sze10] Section 8.4.



Hierarchical motion estimation (with image
pyramid): [Sze10] Section 8.1.1.

CS4243

Motion Tracking

48

References


[Bir] S. Birchfield. KLT: An implementation of the Kanade-LucasTomasi feature tracker. http://vision.stanford.edu/~birch/klt/.



[BK08] Bradski and Kaehler. Learning OpenCV: Computer Vision
with the OpenCV Library. O’Reilly, 2008.



[LK81] B. D. Lucas and T. Kanade. An iterative image registration
technique with an application to stereo vision. In Proceedings of 7th
International Joint Conference on Artificial Intelligence, pages 674–
679, 1981.



[ST94] J. Shi and C. Tomasi. Good features to track. In Proceedings
of IEEE Conference on Computer Vision and Pattern Recognition,
pages 593–600, 1994.



[Sze10] R. Szeliski. Computer Vision: Algorithms and Applications.
Springer, 2010.

CS4243

Motion Tracking

49

References


CS4243

[TK91] C. Tomasi and T. Kanade. Detection and tracking of point
features. Technical Report CMU-CS-91-132, School of Computer
Science, Carnegie Mellon University, 1991.

Motion Tracking

50


Slide 25

Leow Wee Kheng
CS4243 Computer Vision and Pattern Recognition

Motion Tracking

CS4243

Motion Tracking

1

Changes are everywhere!

CS4243

Motion Tracking

2

Illumination change

CS4243

Motion Tracking

3

Shape change

CS4243

Motion Tracking

4

Object motion

CS4243

Motion Tracking

5

Camera motion

CS4243

Motion Tracking

6

Object & camera motion

CS4243

Motion Tracking

7

Everything changes!

CS4243

Motion Tracking

8

Motion analysis is tough in general!

We focus on object / camera motion.

CS4243

Motion Tracking

9

Change Detection


Detects any change in two video frames.



Straightforward method:
 Compute

difference between corresponding pixels:



: intensity / colour at (x, y) in frame t.
> threshold, has large difference.

 If

CS4243

Motion Tracking

10

Any difference?

No

CS4243

Motion Tracking

11

Any difference?

Yes,
illumination
change

CS4243

Motion Tracking

12

Any difference?

Yes,
position change

CS4243

Motion Tracking

13

Change Detection


Can detect
 Illumination

change
 Position change
 Illumination and position change


But, cannot distinguish between them.



Need to detect and measure position change.

CS4243

Motion Tracking

14

Motion Tracking


Two approaches
 Feature-based
 Intensity

CS4243

gradient-based

Motion Tracking

15

Feature-based Motion Tracking


Look for distinct features that change positions.

 Eagle’s

wing tips change positions.
 Tree tops don’t change positions.
CS4243

Motion Tracking

16

Basic Ideas
1.

Look for distinct features in current frame.

2.

For each feature


Search for matching feature within neighbourhood
in next frame.
 Difference in positions  displacement.
 Velocity = displacement / time difference.

CS4243

Motion Tracking

17

Basic Ideas
displacement

feature area

search area

CS4243

Motion Tracking

18

What feature to use?


Harris corner



Tomasi’s feature



Feature descriptors: SIFT, SURF, GLOH, etc.



Others

CS4243

Motion Tracking

19

Summary


Simple algorithm.



Can be slow if search area is large.



Can constrain search area with prior knowledge.

CS4243

Motion Tracking

20

Gradient-based Motion Tracking


Two basic assumptions
 Intensity

changes smoothly with position.
 Intensity of object doesn’t change over time.


Suppose an object is in motion.
 Change

position (dx, dy) over time dt.
 Then, from 2nd assumption:
 Apply

CS4243

Taylor’s series expansion:

Motion Tracking

21

 Omit

higher order terms and divide by dt

 Denote

 Then,

 u,

v are unknown.
 2 unknowns, 1 equation: can’t solve!

CS4243

Motion Tracking

22

Any difference in motion?

CS4243

Motion Tracking

23

Any difference in motion?

CS4243

Motion Tracking

24

Any difference in motion?

CS4243

Motion Tracking

25

Any difference in motion?

CS4243

Motion Tracking

26

Any difference in motion?

CS4243

Motion Tracking

27

Any difference in motion?

CS4243

Motion Tracking

28

Aperture Problem


Homogeneous region

 Ix

= Iy = It = 0.
 No change in local region.
 Cannot detect motion.

CS4243

Motion Tracking

29

Aperture Problem


Edge

 Ix

and Iy are zero along edge
 Cannot measure motion tangential to edge
 Ix and Iy are non-zero normal to edge
 Can measure motion normal to edge
 So, cannot measure actual motion

CS4243

Motion Tracking

30

Aperture Problem


Corner

 Ix

and Iy are non-zero in two perpendicular directions
 2 unknowns, 2 equations
 Can measure actual motion

CS4243

Motion Tracking

31

Lucas-Kanade Method


Consider two consecutive image frames I and J:
I

J
x+d
d
x

x

moves from x = (x, y)T to x + d.
 d = (u, v) T
 Object

CS4243

Motion Tracking

32



So,



Or



Due to noise, there’s an error at position x:



Sum error over small window W at position x:

Similar to
template matching
CS4243

weight
Motion Tracking

33



If E is small, patterns in I and J match well.



So, find d that minimises E:
E / d = 0, compute d that minimises E.
 First, expand I(x – d) by Taylor’s series expansion:
 Set

 Omit

higher order terms:

 Write

in matrix form:

Intensity gradient
CS4243

Motion Tracking

34

 Now,

error E at position x is:

 Now,

differentiate E with respect to d (exercise):

 Setting

E / d = 0 gives

b

the only
unknown

Z

CS4243

Motion Tracking

35

 So,

2



CS4243

we get

unknowns, 2 equations. Can solve for d = (u, v).

What happen to aperture problem?
Did it disappear?
Motion Tracking

36

Lucas-Kanade + Tomasi


Lucas-Kanade algorithm is often used with
Tomasi’s feature
 Apply

Tomasi’s method to detect good features.
 Apply LK method to compute d for each pixel.
 Accept d only for good features.

CS4243

Motion Tracking

37

Example

Can you spot tracking errors?

CS4243

Motion Tracking

38

Constraints


Math of LK tracker assumes d is small.



In implementation, W is also small.



LK tracker is good only for small displacement.

CS4243

Motion Tracking

39

How to handle large displacement?


What if we scale down images?
 Displacements

CS4243

are smaller!

Motion Tracking

40

Image Pyramid
Usually, smoothen
image with Gaussian
filter before scaling
down.

downsample

scaled image

input image
CS4243

Motion Tracking

41

LK Tracking with Image Pyramid

CS4243

Motion Tracking

42



Construct image pyramids.



Apply LK tracker to low-resolution images.



Propagate results to higher-resolution images.



Apply LK tracker to higher-resolution images.

CS4243

Motion Tracking

43

Example

Tracking results are more accurate.

CS4243

Motion Tracking

44

Summary


Efficient algorithm, no explicit search.



Has aperture problem;
track good features only



LK tracker can’t track large displacement.



Use LK + image pyramid for large displacement.

CS4243

Motion Tracking

45

Software


OpenCV supports LK and LK with pyramid.



[Bir] offers LK with Tomasi’s features & pyramid.

CS4243

Motion Tracking

46

Appendix


Calculation of Ix, Iy, It
 Use

finite difference method
 Forward difference
Ix = I(x  1, y, t)  I(x, y, t)
Iy = I(x, y  1, t)  I(x, y, t)
It = I(x, y, t  1)  I(x, y, t)
 Backward difference
Ix = I(x, y, t)  I(x  1, y, t)
Iy = I(x, y, t)  I(x, y  1, t)
It = I(x, y, t)  I(x, y, t  1)

CS4243

Motion Tracking

47

Further Readings


Lucas-Kanade tracking with pyramid:
[BK08] Chapter 10.



Optical flow: [Sze10] Section 8.4.



Hierarchical motion estimation (with image
pyramid): [Sze10] Section 8.1.1.

CS4243

Motion Tracking

48

References


[Bir] S. Birchfield. KLT: An implementation of the Kanade-LucasTomasi feature tracker. http://vision.stanford.edu/~birch/klt/.



[BK08] Bradski and Kaehler. Learning OpenCV: Computer Vision
with the OpenCV Library. O’Reilly, 2008.



[LK81] B. D. Lucas and T. Kanade. An iterative image registration
technique with an application to stereo vision. In Proceedings of 7th
International Joint Conference on Artificial Intelligence, pages 674–
679, 1981.



[ST94] J. Shi and C. Tomasi. Good features to track. In Proceedings
of IEEE Conference on Computer Vision and Pattern Recognition,
pages 593–600, 1994.



[Sze10] R. Szeliski. Computer Vision: Algorithms and Applications.
Springer, 2010.

CS4243

Motion Tracking

49

References


CS4243

[TK91] C. Tomasi and T. Kanade. Detection and tracking of point
features. Technical Report CMU-CS-91-132, School of Computer
Science, Carnegie Mellon University, 1991.

Motion Tracking

50


Slide 26

Leow Wee Kheng
CS4243 Computer Vision and Pattern Recognition

Motion Tracking

CS4243

Motion Tracking

1

Changes are everywhere!

CS4243

Motion Tracking

2

Illumination change

CS4243

Motion Tracking

3

Shape change

CS4243

Motion Tracking

4

Object motion

CS4243

Motion Tracking

5

Camera motion

CS4243

Motion Tracking

6

Object & camera motion

CS4243

Motion Tracking

7

Everything changes!

CS4243

Motion Tracking

8

Motion analysis is tough in general!

We focus on object / camera motion.

CS4243

Motion Tracking

9

Change Detection


Detects any change in two video frames.



Straightforward method:
 Compute

difference between corresponding pixels:



: intensity / colour at (x, y) in frame t.
> threshold, has large difference.

 If

CS4243

Motion Tracking

10

Any difference?

No

CS4243

Motion Tracking

11

Any difference?

Yes,
illumination
change

CS4243

Motion Tracking

12

Any difference?

Yes,
position change

CS4243

Motion Tracking

13

Change Detection


Can detect
 Illumination

change
 Position change
 Illumination and position change


But, cannot distinguish between them.



Need to detect and measure position change.

CS4243

Motion Tracking

14

Motion Tracking


Two approaches
 Feature-based
 Intensity

CS4243

gradient-based

Motion Tracking

15

Feature-based Motion Tracking


Look for distinct features that change positions.

 Eagle’s

wing tips change positions.
 Tree tops don’t change positions.
CS4243

Motion Tracking

16

Basic Ideas
1.

Look for distinct features in current frame.

2.

For each feature


Search for matching feature within neighbourhood
in next frame.
 Difference in positions  displacement.
 Velocity = displacement / time difference.

CS4243

Motion Tracking

17

Basic Ideas
displacement

feature area

search area

CS4243

Motion Tracking

18

What feature to use?


Harris corner



Tomasi’s feature



Feature descriptors: SIFT, SURF, GLOH, etc.



Others

CS4243

Motion Tracking

19

Summary


Simple algorithm.



Can be slow if search area is large.



Can constrain search area with prior knowledge.

CS4243

Motion Tracking

20

Gradient-based Motion Tracking


Two basic assumptions
 Intensity

changes smoothly with position.
 Intensity of object doesn’t change over time.


Suppose an object is in motion.
 Change

position (dx, dy) over time dt.
 Then, from 2nd assumption:
 Apply

CS4243

Taylor’s series expansion:

Motion Tracking

21

 Omit

higher order terms and divide by dt

 Denote

 Then,

 u,

v are unknown.
 2 unknowns, 1 equation: can’t solve!

CS4243

Motion Tracking

22

Any difference in motion?

CS4243

Motion Tracking

23

Any difference in motion?

CS4243

Motion Tracking

24

Any difference in motion?

CS4243

Motion Tracking

25

Any difference in motion?

CS4243

Motion Tracking

26

Any difference in motion?

CS4243

Motion Tracking

27

Any difference in motion?

CS4243

Motion Tracking

28

Aperture Problem


Homogeneous region

 Ix

= Iy = It = 0.
 No change in local region.
 Cannot detect motion.

CS4243

Motion Tracking

29

Aperture Problem


Edge

 Ix

and Iy are zero along edge
 Cannot measure motion tangential to edge
 Ix and Iy are non-zero normal to edge
 Can measure motion normal to edge
 So, cannot measure actual motion

CS4243

Motion Tracking

30

Aperture Problem


Corner

 Ix

and Iy are non-zero in two perpendicular directions
 2 unknowns, 2 equations
 Can measure actual motion

CS4243

Motion Tracking

31

Lucas-Kanade Method


Consider two consecutive image frames I and J:
I

J
x+d
d
x

x

moves from x = (x, y)T to x + d.
 d = (u, v) T
 Object

CS4243

Motion Tracking

32



So,



Or



Due to noise, there’s an error at position x:



Sum error over small window W at position x:

Similar to
template matching
CS4243

weight
Motion Tracking

33



If E is small, patterns in I and J match well.



So, find d that minimises E:
E / d = 0, compute d that minimises E.
 First, expand I(x – d) by Taylor’s series expansion:
 Set

 Omit

higher order terms:

 Write

in matrix form:

Intensity gradient
CS4243

Motion Tracking

34

 Now,

error E at position x is:

 Now,

differentiate E with respect to d (exercise):

 Setting

E / d = 0 gives

b

the only
unknown

Z

CS4243

Motion Tracking

35

 So,

2



CS4243

we get

unknowns, 2 equations. Can solve for d = (u, v).

What happen to aperture problem?
Did it disappear?
Motion Tracking

36

Lucas-Kanade + Tomasi


Lucas-Kanade algorithm is often used with
Tomasi’s feature
 Apply

Tomasi’s method to detect good features.
 Apply LK method to compute d for each pixel.
 Accept d only for good features.

CS4243

Motion Tracking

37

Example

Can you spot tracking errors?

CS4243

Motion Tracking

38

Constraints


Math of LK tracker assumes d is small.



In implementation, W is also small.



LK tracker is good only for small displacement.

CS4243

Motion Tracking

39

How to handle large displacement?


What if we scale down images?
 Displacements

CS4243

are smaller!

Motion Tracking

40

Image Pyramid
Usually, smoothen
image with Gaussian
filter before scaling
down.

downsample

scaled image

input image
CS4243

Motion Tracking

41

LK Tracking with Image Pyramid

CS4243

Motion Tracking

42



Construct image pyramids.



Apply LK tracker to low-resolution images.



Propagate results to higher-resolution images.



Apply LK tracker to higher-resolution images.

CS4243

Motion Tracking

43

Example

Tracking results are more accurate.

CS4243

Motion Tracking

44

Summary


Efficient algorithm, no explicit search.



Has aperture problem;
track good features only



LK tracker can’t track large displacement.



Use LK + image pyramid for large displacement.

CS4243

Motion Tracking

45

Software


OpenCV supports LK and LK with pyramid.



[Bir] offers LK with Tomasi’s features & pyramid.

CS4243

Motion Tracking

46

Appendix


Calculation of Ix, Iy, It
 Use

finite difference method
 Forward difference
Ix = I(x  1, y, t)  I(x, y, t)
Iy = I(x, y  1, t)  I(x, y, t)
It = I(x, y, t  1)  I(x, y, t)
 Backward difference
Ix = I(x, y, t)  I(x  1, y, t)
Iy = I(x, y, t)  I(x, y  1, t)
It = I(x, y, t)  I(x, y, t  1)

CS4243

Motion Tracking

47

Further Readings


Lucas-Kanade tracking with pyramid:
[BK08] Chapter 10.



Optical flow: [Sze10] Section 8.4.



Hierarchical motion estimation (with image
pyramid): [Sze10] Section 8.1.1.

CS4243

Motion Tracking

48

References


[Bir] S. Birchfield. KLT: An implementation of the Kanade-LucasTomasi feature tracker. http://vision.stanford.edu/~birch/klt/.



[BK08] Bradski and Kaehler. Learning OpenCV: Computer Vision
with the OpenCV Library. O’Reilly, 2008.



[LK81] B. D. Lucas and T. Kanade. An iterative image registration
technique with an application to stereo vision. In Proceedings of 7th
International Joint Conference on Artificial Intelligence, pages 674–
679, 1981.



[ST94] J. Shi and C. Tomasi. Good features to track. In Proceedings
of IEEE Conference on Computer Vision and Pattern Recognition,
pages 593–600, 1994.



[Sze10] R. Szeliski. Computer Vision: Algorithms and Applications.
Springer, 2010.

CS4243

Motion Tracking

49

References


CS4243

[TK91] C. Tomasi and T. Kanade. Detection and tracking of point
features. Technical Report CMU-CS-91-132, School of Computer
Science, Carnegie Mellon University, 1991.

Motion Tracking

50


Slide 27

Leow Wee Kheng
CS4243 Computer Vision and Pattern Recognition

Motion Tracking

CS4243

Motion Tracking

1

Changes are everywhere!

CS4243

Motion Tracking

2

Illumination change

CS4243

Motion Tracking

3

Shape change

CS4243

Motion Tracking

4

Object motion

CS4243

Motion Tracking

5

Camera motion

CS4243

Motion Tracking

6

Object & camera motion

CS4243

Motion Tracking

7

Everything changes!

CS4243

Motion Tracking

8

Motion analysis is tough in general!

We focus on object / camera motion.

CS4243

Motion Tracking

9

Change Detection


Detects any change in two video frames.



Straightforward method:
 Compute

difference between corresponding pixels:



: intensity / colour at (x, y) in frame t.
> threshold, has large difference.

 If

CS4243

Motion Tracking

10

Any difference?

No

CS4243

Motion Tracking

11

Any difference?

Yes,
illumination
change

CS4243

Motion Tracking

12

Any difference?

Yes,
position change

CS4243

Motion Tracking

13

Change Detection


Can detect
 Illumination

change
 Position change
 Illumination and position change


But, cannot distinguish between them.



Need to detect and measure position change.

CS4243

Motion Tracking

14

Motion Tracking


Two approaches
 Feature-based
 Intensity

CS4243

gradient-based

Motion Tracking

15

Feature-based Motion Tracking


Look for distinct features that change positions.

 Eagle’s

wing tips change positions.
 Tree tops don’t change positions.
CS4243

Motion Tracking

16

Basic Ideas
1.

Look for distinct features in current frame.

2.

For each feature


Search for matching feature within neighbourhood
in next frame.
 Difference in positions  displacement.
 Velocity = displacement / time difference.

CS4243

Motion Tracking

17

Basic Ideas
displacement

feature area

search area

CS4243

Motion Tracking

18

What feature to use?


Harris corner



Tomasi’s feature



Feature descriptors: SIFT, SURF, GLOH, etc.



Others

CS4243

Motion Tracking

19

Summary


Simple algorithm.



Can be slow if search area is large.



Can constrain search area with prior knowledge.

CS4243

Motion Tracking

20

Gradient-based Motion Tracking


Two basic assumptions
 Intensity

changes smoothly with position.
 Intensity of object doesn’t change over time.


Suppose an object is in motion.
 Change

position (dx, dy) over time dt.
 Then, from 2nd assumption:
 Apply

CS4243

Taylor’s series expansion:

Motion Tracking

21

 Omit

higher order terms and divide by dt

 Denote

 Then,

 u,

v are unknown.
 2 unknowns, 1 equation: can’t solve!

CS4243

Motion Tracking

22

Any difference in motion?

CS4243

Motion Tracking

23

Any difference in motion?

CS4243

Motion Tracking

24

Any difference in motion?

CS4243

Motion Tracking

25

Any difference in motion?

CS4243

Motion Tracking

26

Any difference in motion?

CS4243

Motion Tracking

27

Any difference in motion?

CS4243

Motion Tracking

28

Aperture Problem


Homogeneous region

 Ix

= Iy = It = 0.
 No change in local region.
 Cannot detect motion.

CS4243

Motion Tracking

29

Aperture Problem


Edge

 Ix

and Iy are zero along edge
 Cannot measure motion tangential to edge
 Ix and Iy are non-zero normal to edge
 Can measure motion normal to edge
 So, cannot measure actual motion

CS4243

Motion Tracking

30

Aperture Problem


Corner

 Ix

and Iy are non-zero in two perpendicular directions
 2 unknowns, 2 equations
 Can measure actual motion

CS4243

Motion Tracking

31

Lucas-Kanade Method


Consider two consecutive image frames I and J:
I

J
x+d
d
x

x

moves from x = (x, y)T to x + d.
 d = (u, v) T
 Object

CS4243

Motion Tracking

32



So,



Or



Due to noise, there’s an error at position x:



Sum error over small window W at position x:

Similar to
template matching
CS4243

weight
Motion Tracking

33



If E is small, patterns in I and J match well.



So, find d that minimises E:
E / d = 0, compute d that minimises E.
 First, expand I(x – d) by Taylor’s series expansion:
 Set

 Omit

higher order terms:

 Write

in matrix form:

Intensity gradient
CS4243

Motion Tracking

34

 Now,

error E at position x is:

 Now,

differentiate E with respect to d (exercise):

 Setting

E / d = 0 gives

b

the only
unknown

Z

CS4243

Motion Tracking

35

 So,

2



CS4243

we get

unknowns, 2 equations. Can solve for d = (u, v).

What happen to aperture problem?
Did it disappear?
Motion Tracking

36

Lucas-Kanade + Tomasi


Lucas-Kanade algorithm is often used with
Tomasi’s feature
 Apply

Tomasi’s method to detect good features.
 Apply LK method to compute d for each pixel.
 Accept d only for good features.

CS4243

Motion Tracking

37

Example

Can you spot tracking errors?

CS4243

Motion Tracking

38

Constraints


Math of LK tracker assumes d is small.



In implementation, W is also small.



LK tracker is good only for small displacement.

CS4243

Motion Tracking

39

How to handle large displacement?


What if we scale down images?
 Displacements

CS4243

are smaller!

Motion Tracking

40

Image Pyramid
Usually, smoothen
image with Gaussian
filter before scaling
down.

downsample

scaled image

input image
CS4243

Motion Tracking

41

LK Tracking with Image Pyramid

CS4243

Motion Tracking

42



Construct image pyramids.



Apply LK tracker to low-resolution images.



Propagate results to higher-resolution images.



Apply LK tracker to higher-resolution images.

CS4243

Motion Tracking

43

Example

Tracking results are more accurate.

CS4243

Motion Tracking

44

Summary


Efficient algorithm, no explicit search.



Has aperture problem;
track good features only



LK tracker can’t track large displacement.



Use LK + image pyramid for large displacement.

CS4243

Motion Tracking

45

Software


OpenCV supports LK and LK with pyramid.



[Bir] offers LK with Tomasi’s features & pyramid.

CS4243

Motion Tracking

46

Appendix


Calculation of Ix, Iy, It
 Use

finite difference method
 Forward difference
Ix = I(x  1, y, t)  I(x, y, t)
Iy = I(x, y  1, t)  I(x, y, t)
It = I(x, y, t  1)  I(x, y, t)
 Backward difference
Ix = I(x, y, t)  I(x  1, y, t)
Iy = I(x, y, t)  I(x, y  1, t)
It = I(x, y, t)  I(x, y, t  1)

CS4243

Motion Tracking

47

Further Readings


Lucas-Kanade tracking with pyramid:
[BK08] Chapter 10.



Optical flow: [Sze10] Section 8.4.



Hierarchical motion estimation (with image
pyramid): [Sze10] Section 8.1.1.

CS4243

Motion Tracking

48

References


[Bir] S. Birchfield. KLT: An implementation of the Kanade-LucasTomasi feature tracker. http://vision.stanford.edu/~birch/klt/.



[BK08] Bradski and Kaehler. Learning OpenCV: Computer Vision
with the OpenCV Library. O’Reilly, 2008.



[LK81] B. D. Lucas and T. Kanade. An iterative image registration
technique with an application to stereo vision. In Proceedings of 7th
International Joint Conference on Artificial Intelligence, pages 674–
679, 1981.



[ST94] J. Shi and C. Tomasi. Good features to track. In Proceedings
of IEEE Conference on Computer Vision and Pattern Recognition,
pages 593–600, 1994.



[Sze10] R. Szeliski. Computer Vision: Algorithms and Applications.
Springer, 2010.

CS4243

Motion Tracking

49

References


CS4243

[TK91] C. Tomasi and T. Kanade. Detection and tracking of point
features. Technical Report CMU-CS-91-132, School of Computer
Science, Carnegie Mellon University, 1991.

Motion Tracking

50


Slide 28

Leow Wee Kheng
CS4243 Computer Vision and Pattern Recognition

Motion Tracking

CS4243

Motion Tracking

1

Changes are everywhere!

CS4243

Motion Tracking

2

Illumination change

CS4243

Motion Tracking

3

Shape change

CS4243

Motion Tracking

4

Object motion

CS4243

Motion Tracking

5

Camera motion

CS4243

Motion Tracking

6

Object & camera motion

CS4243

Motion Tracking

7

Everything changes!

CS4243

Motion Tracking

8

Motion analysis is tough in general!

We focus on object / camera motion.

CS4243

Motion Tracking

9

Change Detection


Detects any change in two video frames.



Straightforward method:
 Compute

difference between corresponding pixels:



: intensity / colour at (x, y) in frame t.
> threshold, has large difference.

 If

CS4243

Motion Tracking

10

Any difference?

No

CS4243

Motion Tracking

11

Any difference?

Yes,
illumination
change

CS4243

Motion Tracking

12

Any difference?

Yes,
position change

CS4243

Motion Tracking

13

Change Detection


Can detect
 Illumination

change
 Position change
 Illumination and position change


But, cannot distinguish between them.



Need to detect and measure position change.

CS4243

Motion Tracking

14

Motion Tracking


Two approaches
 Feature-based
 Intensity

CS4243

gradient-based

Motion Tracking

15

Feature-based Motion Tracking


Look for distinct features that change positions.

 Eagle’s

wing tips change positions.
 Tree tops don’t change positions.
CS4243

Motion Tracking

16

Basic Ideas
1.

Look for distinct features in current frame.

2.

For each feature


Search for matching feature within neighbourhood
in next frame.
 Difference in positions  displacement.
 Velocity = displacement / time difference.

CS4243

Motion Tracking

17

Basic Ideas
displacement

feature area

search area

CS4243

Motion Tracking

18

What feature to use?


Harris corner



Tomasi’s feature



Feature descriptors: SIFT, SURF, GLOH, etc.



Others

CS4243

Motion Tracking

19

Summary


Simple algorithm.



Can be slow if search area is large.



Can constrain search area with prior knowledge.

CS4243

Motion Tracking

20

Gradient-based Motion Tracking


Two basic assumptions
 Intensity

changes smoothly with position.
 Intensity of object doesn’t change over time.


Suppose an object is in motion.
 Change

position (dx, dy) over time dt.
 Then, from 2nd assumption:
 Apply

CS4243

Taylor’s series expansion:

Motion Tracking

21

 Omit

higher order terms and divide by dt

 Denote

 Then,

 u,

v are unknown.
 2 unknowns, 1 equation: can’t solve!

CS4243

Motion Tracking

22

Any difference in motion?

CS4243

Motion Tracking

23

Any difference in motion?

CS4243

Motion Tracking

24

Any difference in motion?

CS4243

Motion Tracking

25

Any difference in motion?

CS4243

Motion Tracking

26

Any difference in motion?

CS4243

Motion Tracking

27

Any difference in motion?

CS4243

Motion Tracking

28

Aperture Problem


Homogeneous region

 Ix

= Iy = It = 0.
 No change in local region.
 Cannot detect motion.

CS4243

Motion Tracking

29

Aperture Problem


Edge

 Ix

and Iy are zero along edge
 Cannot measure motion tangential to edge
 Ix and Iy are non-zero normal to edge
 Can measure motion normal to edge
 So, cannot measure actual motion

CS4243

Motion Tracking

30

Aperture Problem


Corner

 Ix

and Iy are non-zero in two perpendicular directions
 2 unknowns, 2 equations
 Can measure actual motion

CS4243

Motion Tracking

31

Lucas-Kanade Method


Consider two consecutive image frames I and J:
I

J
x+d
d
x

x

moves from x = (x, y)T to x + d.
 d = (u, v) T
 Object

CS4243

Motion Tracking

32



So,



Or



Due to noise, there’s an error at position x:



Sum error over small window W at position x:

Similar to
template matching
CS4243

weight
Motion Tracking

33



If E is small, patterns in I and J match well.



So, find d that minimises E:
E / d = 0, compute d that minimises E.
 First, expand I(x – d) by Taylor’s series expansion:
 Set

 Omit

higher order terms:

 Write

in matrix form:

Intensity gradient
CS4243

Motion Tracking

34

 Now,

error E at position x is:

 Now,

differentiate E with respect to d (exercise):

 Setting

E / d = 0 gives

b

the only
unknown

Z

CS4243

Motion Tracking

35

 So,

2



CS4243

we get

unknowns, 2 equations. Can solve for d = (u, v).

What happen to aperture problem?
Did it disappear?
Motion Tracking

36

Lucas-Kanade + Tomasi


Lucas-Kanade algorithm is often used with
Tomasi’s feature
 Apply

Tomasi’s method to detect good features.
 Apply LK method to compute d for each pixel.
 Accept d only for good features.

CS4243

Motion Tracking

37

Example

Can you spot tracking errors?

CS4243

Motion Tracking

38

Constraints


Math of LK tracker assumes d is small.



In implementation, W is also small.



LK tracker is good only for small displacement.

CS4243

Motion Tracking

39

How to handle large displacement?


What if we scale down images?
 Displacements

CS4243

are smaller!

Motion Tracking

40

Image Pyramid
Usually, smoothen
image with Gaussian
filter before scaling
down.

downsample

scaled image

input image
CS4243

Motion Tracking

41

LK Tracking with Image Pyramid

CS4243

Motion Tracking

42



Construct image pyramids.



Apply LK tracker to low-resolution images.



Propagate results to higher-resolution images.



Apply LK tracker to higher-resolution images.

CS4243

Motion Tracking

43

Example

Tracking results are more accurate.

CS4243

Motion Tracking

44

Summary


Efficient algorithm, no explicit search.



Has aperture problem;
track good features only



LK tracker can’t track large displacement.



Use LK + image pyramid for large displacement.

CS4243

Motion Tracking

45

Software


OpenCV supports LK and LK with pyramid.



[Bir] offers LK with Tomasi’s features & pyramid.

CS4243

Motion Tracking

46

Appendix


Calculation of Ix, Iy, It
 Use

finite difference method
 Forward difference
Ix = I(x  1, y, t)  I(x, y, t)
Iy = I(x, y  1, t)  I(x, y, t)
It = I(x, y, t  1)  I(x, y, t)
 Backward difference
Ix = I(x, y, t)  I(x  1, y, t)
Iy = I(x, y, t)  I(x, y  1, t)
It = I(x, y, t)  I(x, y, t  1)

CS4243

Motion Tracking

47

Further Readings


Lucas-Kanade tracking with pyramid:
[BK08] Chapter 10.



Optical flow: [Sze10] Section 8.4.



Hierarchical motion estimation (with image
pyramid): [Sze10] Section 8.1.1.

CS4243

Motion Tracking

48

References


[Bir] S. Birchfield. KLT: An implementation of the Kanade-LucasTomasi feature tracker. http://vision.stanford.edu/~birch/klt/.



[BK08] Bradski and Kaehler. Learning OpenCV: Computer Vision
with the OpenCV Library. O’Reilly, 2008.



[LK81] B. D. Lucas and T. Kanade. An iterative image registration
technique with an application to stereo vision. In Proceedings of 7th
International Joint Conference on Artificial Intelligence, pages 674–
679, 1981.



[ST94] J. Shi and C. Tomasi. Good features to track. In Proceedings
of IEEE Conference on Computer Vision and Pattern Recognition,
pages 593–600, 1994.



[Sze10] R. Szeliski. Computer Vision: Algorithms and Applications.
Springer, 2010.

CS4243

Motion Tracking

49

References


CS4243

[TK91] C. Tomasi and T. Kanade. Detection and tracking of point
features. Technical Report CMU-CS-91-132, School of Computer
Science, Carnegie Mellon University, 1991.

Motion Tracking

50


Slide 29

Leow Wee Kheng
CS4243 Computer Vision and Pattern Recognition

Motion Tracking

CS4243

Motion Tracking

1

Changes are everywhere!

CS4243

Motion Tracking

2

Illumination change

CS4243

Motion Tracking

3

Shape change

CS4243

Motion Tracking

4

Object motion

CS4243

Motion Tracking

5

Camera motion

CS4243

Motion Tracking

6

Object & camera motion

CS4243

Motion Tracking

7

Everything changes!

CS4243

Motion Tracking

8

Motion analysis is tough in general!

We focus on object / camera motion.

CS4243

Motion Tracking

9

Change Detection


Detects any change in two video frames.



Straightforward method:
 Compute

difference between corresponding pixels:



: intensity / colour at (x, y) in frame t.
> threshold, has large difference.

 If

CS4243

Motion Tracking

10

Any difference?

No

CS4243

Motion Tracking

11

Any difference?

Yes,
illumination
change

CS4243

Motion Tracking

12

Any difference?

Yes,
position change

CS4243

Motion Tracking

13

Change Detection


Can detect
 Illumination

change
 Position change
 Illumination and position change


But, cannot distinguish between them.



Need to detect and measure position change.

CS4243

Motion Tracking

14

Motion Tracking


Two approaches
 Feature-based
 Intensity

CS4243

gradient-based

Motion Tracking

15

Feature-based Motion Tracking


Look for distinct features that change positions.

 Eagle’s

wing tips change positions.
 Tree tops don’t change positions.
CS4243

Motion Tracking

16

Basic Ideas
1.

Look for distinct features in current frame.

2.

For each feature


Search for matching feature within neighbourhood
in next frame.
 Difference in positions  displacement.
 Velocity = displacement / time difference.

CS4243

Motion Tracking

17

Basic Ideas
displacement

feature area

search area

CS4243

Motion Tracking

18

What feature to use?


Harris corner



Tomasi’s feature



Feature descriptors: SIFT, SURF, GLOH, etc.



Others

CS4243

Motion Tracking

19

Summary


Simple algorithm.



Can be slow if search area is large.



Can constrain search area with prior knowledge.

CS4243

Motion Tracking

20

Gradient-based Motion Tracking


Two basic assumptions
 Intensity

changes smoothly with position.
 Intensity of object doesn’t change over time.


Suppose an object is in motion.
 Change

position (dx, dy) over time dt.
 Then, from 2nd assumption:
 Apply

CS4243

Taylor’s series expansion:

Motion Tracking

21

 Omit

higher order terms and divide by dt

 Denote

 Then,

 u,

v are unknown.
 2 unknowns, 1 equation: can’t solve!

CS4243

Motion Tracking

22

Any difference in motion?

CS4243

Motion Tracking

23

Any difference in motion?

CS4243

Motion Tracking

24

Any difference in motion?

CS4243

Motion Tracking

25

Any difference in motion?

CS4243

Motion Tracking

26

Any difference in motion?

CS4243

Motion Tracking

27

Any difference in motion?

CS4243

Motion Tracking

28

Aperture Problem


Homogeneous region

 Ix

= Iy = It = 0.
 No change in local region.
 Cannot detect motion.

CS4243

Motion Tracking

29

Aperture Problem


Edge

 Ix

and Iy are zero along edge
 Cannot measure motion tangential to edge
 Ix and Iy are non-zero normal to edge
 Can measure motion normal to edge
 So, cannot measure actual motion

CS4243

Motion Tracking

30

Aperture Problem


Corner

 Ix

and Iy are non-zero in two perpendicular directions
 2 unknowns, 2 equations
 Can measure actual motion

CS4243

Motion Tracking

31

Lucas-Kanade Method


Consider two consecutive image frames I and J:
I

J
x+d
d
x

x

moves from x = (x, y)T to x + d.
 d = (u, v) T
 Object

CS4243

Motion Tracking

32



So,



Or



Due to noise, there’s an error at position x:



Sum error over small window W at position x:

Similar to
template matching
CS4243

weight
Motion Tracking

33



If E is small, patterns in I and J match well.



So, find d that minimises E:
E / d = 0, compute d that minimises E.
 First, expand I(x – d) by Taylor’s series expansion:
 Set

 Omit

higher order terms:

 Write

in matrix form:

Intensity gradient
CS4243

Motion Tracking

34

 Now,

error E at position x is:

 Now,

differentiate E with respect to d (exercise):

 Setting

E / d = 0 gives

b

the only
unknown

Z

CS4243

Motion Tracking

35

 So,

2



CS4243

we get

unknowns, 2 equations. Can solve for d = (u, v).

What happen to aperture problem?
Did it disappear?
Motion Tracking

36

Lucas-Kanade + Tomasi


Lucas-Kanade algorithm is often used with
Tomasi’s feature
 Apply

Tomasi’s method to detect good features.
 Apply LK method to compute d for each pixel.
 Accept d only for good features.

CS4243

Motion Tracking

37

Example

Can you spot tracking errors?

CS4243

Motion Tracking

38

Constraints


Math of LK tracker assumes d is small.



In implementation, W is also small.



LK tracker is good only for small displacement.

CS4243

Motion Tracking

39

How to handle large displacement?


What if we scale down images?
 Displacements

CS4243

are smaller!

Motion Tracking

40

Image Pyramid
Usually, smoothen
image with Gaussian
filter before scaling
down.

downsample

scaled image

input image
CS4243

Motion Tracking

41

LK Tracking with Image Pyramid

CS4243

Motion Tracking

42



Construct image pyramids.



Apply LK tracker to low-resolution images.



Propagate results to higher-resolution images.



Apply LK tracker to higher-resolution images.

CS4243

Motion Tracking

43

Example

Tracking results are more accurate.

CS4243

Motion Tracking

44

Summary


Efficient algorithm, no explicit search.



Has aperture problem;
track good features only



LK tracker can’t track large displacement.



Use LK + image pyramid for large displacement.

CS4243

Motion Tracking

45

Software


OpenCV supports LK and LK with pyramid.



[Bir] offers LK with Tomasi’s features & pyramid.

CS4243

Motion Tracking

46

Appendix


Calculation of Ix, Iy, It
 Use

finite difference method
 Forward difference
Ix = I(x  1, y, t)  I(x, y, t)
Iy = I(x, y  1, t)  I(x, y, t)
It = I(x, y, t  1)  I(x, y, t)
 Backward difference
Ix = I(x, y, t)  I(x  1, y, t)
Iy = I(x, y, t)  I(x, y  1, t)
It = I(x, y, t)  I(x, y, t  1)

CS4243

Motion Tracking

47

Further Readings


Lucas-Kanade tracking with pyramid:
[BK08] Chapter 10.



Optical flow: [Sze10] Section 8.4.



Hierarchical motion estimation (with image
pyramid): [Sze10] Section 8.1.1.

CS4243

Motion Tracking

48

References


[Bir] S. Birchfield. KLT: An implementation of the Kanade-LucasTomasi feature tracker. http://vision.stanford.edu/~birch/klt/.



[BK08] Bradski and Kaehler. Learning OpenCV: Computer Vision
with the OpenCV Library. O’Reilly, 2008.



[LK81] B. D. Lucas and T. Kanade. An iterative image registration
technique with an application to stereo vision. In Proceedings of 7th
International Joint Conference on Artificial Intelligence, pages 674–
679, 1981.



[ST94] J. Shi and C. Tomasi. Good features to track. In Proceedings
of IEEE Conference on Computer Vision and Pattern Recognition,
pages 593–600, 1994.



[Sze10] R. Szeliski. Computer Vision: Algorithms and Applications.
Springer, 2010.

CS4243

Motion Tracking

49

References


CS4243

[TK91] C. Tomasi and T. Kanade. Detection and tracking of point
features. Technical Report CMU-CS-91-132, School of Computer
Science, Carnegie Mellon University, 1991.

Motion Tracking

50


Slide 30

Leow Wee Kheng
CS4243 Computer Vision and Pattern Recognition

Motion Tracking

CS4243

Motion Tracking

1

Changes are everywhere!

CS4243

Motion Tracking

2

Illumination change

CS4243

Motion Tracking

3

Shape change

CS4243

Motion Tracking

4

Object motion

CS4243

Motion Tracking

5

Camera motion

CS4243

Motion Tracking

6

Object & camera motion

CS4243

Motion Tracking

7

Everything changes!

CS4243

Motion Tracking

8

Motion analysis is tough in general!

We focus on object / camera motion.

CS4243

Motion Tracking

9

Change Detection


Detects any change in two video frames.



Straightforward method:
 Compute

difference between corresponding pixels:



: intensity / colour at (x, y) in frame t.
> threshold, has large difference.

 If

CS4243

Motion Tracking

10

Any difference?

No

CS4243

Motion Tracking

11

Any difference?

Yes,
illumination
change

CS4243

Motion Tracking

12

Any difference?

Yes,
position change

CS4243

Motion Tracking

13

Change Detection


Can detect
 Illumination

change
 Position change
 Illumination and position change


But, cannot distinguish between them.



Need to detect and measure position change.

CS4243

Motion Tracking

14

Motion Tracking


Two approaches
 Feature-based
 Intensity

CS4243

gradient-based

Motion Tracking

15

Feature-based Motion Tracking


Look for distinct features that change positions.

 Eagle’s

wing tips change positions.
 Tree tops don’t change positions.
CS4243

Motion Tracking

16

Basic Ideas
1.

Look for distinct features in current frame.

2.

For each feature


Search for matching feature within neighbourhood
in next frame.
 Difference in positions  displacement.
 Velocity = displacement / time difference.

CS4243

Motion Tracking

17

Basic Ideas
displacement

feature area

search area

CS4243

Motion Tracking

18

What feature to use?


Harris corner



Tomasi’s feature



Feature descriptors: SIFT, SURF, GLOH, etc.



Others

CS4243

Motion Tracking

19

Summary


Simple algorithm.



Can be slow if search area is large.



Can constrain search area with prior knowledge.

CS4243

Motion Tracking

20

Gradient-based Motion Tracking


Two basic assumptions
 Intensity

changes smoothly with position.
 Intensity of object doesn’t change over time.


Suppose an object is in motion.
 Change

position (dx, dy) over time dt.
 Then, from 2nd assumption:
 Apply

CS4243

Taylor’s series expansion:

Motion Tracking

21

 Omit

higher order terms and divide by dt

 Denote

 Then,

 u,

v are unknown.
 2 unknowns, 1 equation: can’t solve!

CS4243

Motion Tracking

22

Any difference in motion?

CS4243

Motion Tracking

23

Any difference in motion?

CS4243

Motion Tracking

24

Any difference in motion?

CS4243

Motion Tracking

25

Any difference in motion?

CS4243

Motion Tracking

26

Any difference in motion?

CS4243

Motion Tracking

27

Any difference in motion?

CS4243

Motion Tracking

28

Aperture Problem


Homogeneous region

 Ix

= Iy = It = 0.
 No change in local region.
 Cannot detect motion.

CS4243

Motion Tracking

29

Aperture Problem


Edge

 Ix

and Iy are zero along edge
 Cannot measure motion tangential to edge
 Ix and Iy are non-zero normal to edge
 Can measure motion normal to edge
 So, cannot measure actual motion

CS4243

Motion Tracking

30

Aperture Problem


Corner

 Ix

and Iy are non-zero in two perpendicular directions
 2 unknowns, 2 equations
 Can measure actual motion

CS4243

Motion Tracking

31

Lucas-Kanade Method


Consider two consecutive image frames I and J:
I

J
x+d
d
x

x

moves from x = (x, y)T to x + d.
 d = (u, v) T
 Object

CS4243

Motion Tracking

32



So,



Or



Due to noise, there’s an error at position x:



Sum error over small window W at position x:

Similar to
template matching
CS4243

weight
Motion Tracking

33



If E is small, patterns in I and J match well.



So, find d that minimises E:
E / d = 0, compute d that minimises E.
 First, expand I(x – d) by Taylor’s series expansion:
 Set

 Omit

higher order terms:

 Write

in matrix form:

Intensity gradient
CS4243

Motion Tracking

34

 Now,

error E at position x is:

 Now,

differentiate E with respect to d (exercise):

 Setting

E / d = 0 gives

b

the only
unknown

Z

CS4243

Motion Tracking

35

 So,

2



CS4243

we get

unknowns, 2 equations. Can solve for d = (u, v).

What happen to aperture problem?
Did it disappear?
Motion Tracking

36

Lucas-Kanade + Tomasi


Lucas-Kanade algorithm is often used with
Tomasi’s feature
 Apply

Tomasi’s method to detect good features.
 Apply LK method to compute d for each pixel.
 Accept d only for good features.

CS4243

Motion Tracking

37

Example

Can you spot tracking errors?

CS4243

Motion Tracking

38

Constraints


Math of LK tracker assumes d is small.



In implementation, W is also small.



LK tracker is good only for small displacement.

CS4243

Motion Tracking

39

How to handle large displacement?


What if we scale down images?
 Displacements

CS4243

are smaller!

Motion Tracking

40

Image Pyramid
Usually, smoothen
image with Gaussian
filter before scaling
down.

downsample

scaled image

input image
CS4243

Motion Tracking

41

LK Tracking with Image Pyramid

CS4243

Motion Tracking

42



Construct image pyramids.



Apply LK tracker to low-resolution images.



Propagate results to higher-resolution images.



Apply LK tracker to higher-resolution images.

CS4243

Motion Tracking

43

Example

Tracking results are more accurate.

CS4243

Motion Tracking

44

Summary


Efficient algorithm, no explicit search.



Has aperture problem;
track good features only



LK tracker can’t track large displacement.



Use LK + image pyramid for large displacement.

CS4243

Motion Tracking

45

Software


OpenCV supports LK and LK with pyramid.



[Bir] offers LK with Tomasi’s features & pyramid.

CS4243

Motion Tracking

46

Appendix


Calculation of Ix, Iy, It
 Use

finite difference method
 Forward difference
Ix = I(x  1, y, t)  I(x, y, t)
Iy = I(x, y  1, t)  I(x, y, t)
It = I(x, y, t  1)  I(x, y, t)
 Backward difference
Ix = I(x, y, t)  I(x  1, y, t)
Iy = I(x, y, t)  I(x, y  1, t)
It = I(x, y, t)  I(x, y, t  1)

CS4243

Motion Tracking

47

Further Readings


Lucas-Kanade tracking with pyramid:
[BK08] Chapter 10.



Optical flow: [Sze10] Section 8.4.



Hierarchical motion estimation (with image
pyramid): [Sze10] Section 8.1.1.

CS4243

Motion Tracking

48

References


[Bir] S. Birchfield. KLT: An implementation of the Kanade-LucasTomasi feature tracker. http://vision.stanford.edu/~birch/klt/.



[BK08] Bradski and Kaehler. Learning OpenCV: Computer Vision
with the OpenCV Library. O’Reilly, 2008.



[LK81] B. D. Lucas and T. Kanade. An iterative image registration
technique with an application to stereo vision. In Proceedings of 7th
International Joint Conference on Artificial Intelligence, pages 674–
679, 1981.



[ST94] J. Shi and C. Tomasi. Good features to track. In Proceedings
of IEEE Conference on Computer Vision and Pattern Recognition,
pages 593–600, 1994.



[Sze10] R. Szeliski. Computer Vision: Algorithms and Applications.
Springer, 2010.

CS4243

Motion Tracking

49

References


CS4243

[TK91] C. Tomasi and T. Kanade. Detection and tracking of point
features. Technical Report CMU-CS-91-132, School of Computer
Science, Carnegie Mellon University, 1991.

Motion Tracking

50


Slide 31

Leow Wee Kheng
CS4243 Computer Vision and Pattern Recognition

Motion Tracking

CS4243

Motion Tracking

1

Changes are everywhere!

CS4243

Motion Tracking

2

Illumination change

CS4243

Motion Tracking

3

Shape change

CS4243

Motion Tracking

4

Object motion

CS4243

Motion Tracking

5

Camera motion

CS4243

Motion Tracking

6

Object & camera motion

CS4243

Motion Tracking

7

Everything changes!

CS4243

Motion Tracking

8

Motion analysis is tough in general!

We focus on object / camera motion.

CS4243

Motion Tracking

9

Change Detection


Detects any change in two video frames.



Straightforward method:
 Compute

difference between corresponding pixels:



: intensity / colour at (x, y) in frame t.
> threshold, has large difference.

 If

CS4243

Motion Tracking

10

Any difference?

No

CS4243

Motion Tracking

11

Any difference?

Yes,
illumination
change

CS4243

Motion Tracking

12

Any difference?

Yes,
position change

CS4243

Motion Tracking

13

Change Detection


Can detect
 Illumination

change
 Position change
 Illumination and position change


But, cannot distinguish between them.



Need to detect and measure position change.

CS4243

Motion Tracking

14

Motion Tracking


Two approaches
 Feature-based
 Intensity

CS4243

gradient-based

Motion Tracking

15

Feature-based Motion Tracking


Look for distinct features that change positions.

 Eagle’s

wing tips change positions.
 Tree tops don’t change positions.
CS4243

Motion Tracking

16

Basic Ideas
1.

Look for distinct features in current frame.

2.

For each feature


Search for matching feature within neighbourhood
in next frame.
 Difference in positions  displacement.
 Velocity = displacement / time difference.

CS4243

Motion Tracking

17

Basic Ideas
displacement

feature area

search area

CS4243

Motion Tracking

18

What feature to use?


Harris corner



Tomasi’s feature



Feature descriptors: SIFT, SURF, GLOH, etc.



Others

CS4243

Motion Tracking

19

Summary


Simple algorithm.



Can be slow if search area is large.



Can constrain search area with prior knowledge.

CS4243

Motion Tracking

20

Gradient-based Motion Tracking


Two basic assumptions
 Intensity

changes smoothly with position.
 Intensity of object doesn’t change over time.


Suppose an object is in motion.
 Change

position (dx, dy) over time dt.
 Then, from 2nd assumption:
 Apply

CS4243

Taylor’s series expansion:

Motion Tracking

21

 Omit

higher order terms and divide by dt

 Denote

 Then,

 u,

v are unknown.
 2 unknowns, 1 equation: can’t solve!

CS4243

Motion Tracking

22

Any difference in motion?

CS4243

Motion Tracking

23

Any difference in motion?

CS4243

Motion Tracking

24

Any difference in motion?

CS4243

Motion Tracking

25

Any difference in motion?

CS4243

Motion Tracking

26

Any difference in motion?

CS4243

Motion Tracking

27

Any difference in motion?

CS4243

Motion Tracking

28

Aperture Problem


Homogeneous region

 Ix

= Iy = It = 0.
 No change in local region.
 Cannot detect motion.

CS4243

Motion Tracking

29

Aperture Problem


Edge

 Ix

and Iy are zero along edge
 Cannot measure motion tangential to edge
 Ix and Iy are non-zero normal to edge
 Can measure motion normal to edge
 So, cannot measure actual motion

CS4243

Motion Tracking

30

Aperture Problem


Corner

 Ix

and Iy are non-zero in two perpendicular directions
 2 unknowns, 2 equations
 Can measure actual motion

CS4243

Motion Tracking

31

Lucas-Kanade Method


Consider two consecutive image frames I and J:
I

J
x+d
d
x

x

moves from x = (x, y)T to x + d.
 d = (u, v) T
 Object

CS4243

Motion Tracking

32



So,



Or



Due to noise, there’s an error at position x:



Sum error over small window W at position x:

Similar to
template matching
CS4243

weight
Motion Tracking

33



If E is small, patterns in I and J match well.



So, find d that minimises E:
E / d = 0, compute d that minimises E.
 First, expand I(x – d) by Taylor’s series expansion:
 Set

 Omit

higher order terms:

 Write

in matrix form:

Intensity gradient
CS4243

Motion Tracking

34

 Now,

error E at position x is:

 Now,

differentiate E with respect to d (exercise):

 Setting

E / d = 0 gives

b

the only
unknown

Z

CS4243

Motion Tracking

35

 So,

2



CS4243

we get

unknowns, 2 equations. Can solve for d = (u, v).

What happen to aperture problem?
Did it disappear?
Motion Tracking

36

Lucas-Kanade + Tomasi


Lucas-Kanade algorithm is often used with
Tomasi’s feature
 Apply

Tomasi’s method to detect good features.
 Apply LK method to compute d for each pixel.
 Accept d only for good features.

CS4243

Motion Tracking

37

Example

Can you spot tracking errors?

CS4243

Motion Tracking

38

Constraints


Math of LK tracker assumes d is small.



In implementation, W is also small.



LK tracker is good only for small displacement.

CS4243

Motion Tracking

39

How to handle large displacement?


What if we scale down images?
 Displacements

CS4243

are smaller!

Motion Tracking

40

Image Pyramid
Usually, smoothen
image with Gaussian
filter before scaling
down.

downsample

scaled image

input image
CS4243

Motion Tracking

41

LK Tracking with Image Pyramid

CS4243

Motion Tracking

42



Construct image pyramids.



Apply LK tracker to low-resolution images.



Propagate results to higher-resolution images.



Apply LK tracker to higher-resolution images.

CS4243

Motion Tracking

43

Example

Tracking results are more accurate.

CS4243

Motion Tracking

44

Summary


Efficient algorithm, no explicit search.



Has aperture problem;
track good features only



LK tracker can’t track large displacement.



Use LK + image pyramid for large displacement.

CS4243

Motion Tracking

45

Software


OpenCV supports LK and LK with pyramid.



[Bir] offers LK with Tomasi’s features & pyramid.

CS4243

Motion Tracking

46

Appendix


Calculation of Ix, Iy, It
 Use

finite difference method
 Forward difference
Ix = I(x  1, y, t)  I(x, y, t)
Iy = I(x, y  1, t)  I(x, y, t)
It = I(x, y, t  1)  I(x, y, t)
 Backward difference
Ix = I(x, y, t)  I(x  1, y, t)
Iy = I(x, y, t)  I(x, y  1, t)
It = I(x, y, t)  I(x, y, t  1)

CS4243

Motion Tracking

47

Further Readings


Lucas-Kanade tracking with pyramid:
[BK08] Chapter 10.



Optical flow: [Sze10] Section 8.4.



Hierarchical motion estimation (with image
pyramid): [Sze10] Section 8.1.1.

CS4243

Motion Tracking

48

References


[Bir] S. Birchfield. KLT: An implementation of the Kanade-LucasTomasi feature tracker. http://vision.stanford.edu/~birch/klt/.



[BK08] Bradski and Kaehler. Learning OpenCV: Computer Vision
with the OpenCV Library. O’Reilly, 2008.



[LK81] B. D. Lucas and T. Kanade. An iterative image registration
technique with an application to stereo vision. In Proceedings of 7th
International Joint Conference on Artificial Intelligence, pages 674–
679, 1981.



[ST94] J. Shi and C. Tomasi. Good features to track. In Proceedings
of IEEE Conference on Computer Vision and Pattern Recognition,
pages 593–600, 1994.



[Sze10] R. Szeliski. Computer Vision: Algorithms and Applications.
Springer, 2010.

CS4243

Motion Tracking

49

References


CS4243

[TK91] C. Tomasi and T. Kanade. Detection and tracking of point
features. Technical Report CMU-CS-91-132, School of Computer
Science, Carnegie Mellon University, 1991.

Motion Tracking

50


Slide 32

Leow Wee Kheng
CS4243 Computer Vision and Pattern Recognition

Motion Tracking

CS4243

Motion Tracking

1

Changes are everywhere!

CS4243

Motion Tracking

2

Illumination change

CS4243

Motion Tracking

3

Shape change

CS4243

Motion Tracking

4

Object motion

CS4243

Motion Tracking

5

Camera motion

CS4243

Motion Tracking

6

Object & camera motion

CS4243

Motion Tracking

7

Everything changes!

CS4243

Motion Tracking

8

Motion analysis is tough in general!

We focus on object / camera motion.

CS4243

Motion Tracking

9

Change Detection


Detects any change in two video frames.



Straightforward method:
 Compute

difference between corresponding pixels:



: intensity / colour at (x, y) in frame t.
> threshold, has large difference.

 If

CS4243

Motion Tracking

10

Any difference?

No

CS4243

Motion Tracking

11

Any difference?

Yes,
illumination
change

CS4243

Motion Tracking

12

Any difference?

Yes,
position change

CS4243

Motion Tracking

13

Change Detection


Can detect
 Illumination

change
 Position change
 Illumination and position change


But, cannot distinguish between them.



Need to detect and measure position change.

CS4243

Motion Tracking

14

Motion Tracking


Two approaches
 Feature-based
 Intensity

CS4243

gradient-based

Motion Tracking

15

Feature-based Motion Tracking


Look for distinct features that change positions.

 Eagle’s

wing tips change positions.
 Tree tops don’t change positions.
CS4243

Motion Tracking

16

Basic Ideas
1.

Look for distinct features in current frame.

2.

For each feature


Search for matching feature within neighbourhood
in next frame.
 Difference in positions  displacement.
 Velocity = displacement / time difference.

CS4243

Motion Tracking

17

Basic Ideas
displacement

feature area

search area

CS4243

Motion Tracking

18

What feature to use?


Harris corner



Tomasi’s feature



Feature descriptors: SIFT, SURF, GLOH, etc.



Others

CS4243

Motion Tracking

19

Summary


Simple algorithm.



Can be slow if search area is large.



Can constrain search area with prior knowledge.

CS4243

Motion Tracking

20

Gradient-based Motion Tracking


Two basic assumptions
 Intensity

changes smoothly with position.
 Intensity of object doesn’t change over time.


Suppose an object is in motion.
 Change

position (dx, dy) over time dt.
 Then, from 2nd assumption:
 Apply

CS4243

Taylor’s series expansion:

Motion Tracking

21

 Omit

higher order terms and divide by dt

 Denote

 Then,

 u,

v are unknown.
 2 unknowns, 1 equation: can’t solve!

CS4243

Motion Tracking

22

Any difference in motion?

CS4243

Motion Tracking

23

Any difference in motion?

CS4243

Motion Tracking

24

Any difference in motion?

CS4243

Motion Tracking

25

Any difference in motion?

CS4243

Motion Tracking

26

Any difference in motion?

CS4243

Motion Tracking

27

Any difference in motion?

CS4243

Motion Tracking

28

Aperture Problem


Homogeneous region

 Ix

= Iy = It = 0.
 No change in local region.
 Cannot detect motion.

CS4243

Motion Tracking

29

Aperture Problem


Edge

 Ix

and Iy are zero along edge
 Cannot measure motion tangential to edge
 Ix and Iy are non-zero normal to edge
 Can measure motion normal to edge
 So, cannot measure actual motion

CS4243

Motion Tracking

30

Aperture Problem


Corner

 Ix

and Iy are non-zero in two perpendicular directions
 2 unknowns, 2 equations
 Can measure actual motion

CS4243

Motion Tracking

31

Lucas-Kanade Method


Consider two consecutive image frames I and J:
I

J
x+d
d
x

x

moves from x = (x, y)T to x + d.
 d = (u, v) T
 Object

CS4243

Motion Tracking

32



So,



Or



Due to noise, there’s an error at position x:



Sum error over small window W at position x:

Similar to
template matching
CS4243

weight
Motion Tracking

33



If E is small, patterns in I and J match well.



So, find d that minimises E:
E / d = 0, compute d that minimises E.
 First, expand I(x – d) by Taylor’s series expansion:
 Set

 Omit

higher order terms:

 Write

in matrix form:

Intensity gradient
CS4243

Motion Tracking

34

 Now,

error E at position x is:

 Now,

differentiate E with respect to d (exercise):

 Setting

E / d = 0 gives

b

the only
unknown

Z

CS4243

Motion Tracking

35

 So,

2



CS4243

we get

unknowns, 2 equations. Can solve for d = (u, v).

What happen to aperture problem?
Did it disappear?
Motion Tracking

36

Lucas-Kanade + Tomasi


Lucas-Kanade algorithm is often used with
Tomasi’s feature
 Apply

Tomasi’s method to detect good features.
 Apply LK method to compute d for each pixel.
 Accept d only for good features.

CS4243

Motion Tracking

37

Example

Can you spot tracking errors?

CS4243

Motion Tracking

38

Constraints


Math of LK tracker assumes d is small.



In implementation, W is also small.



LK tracker is good only for small displacement.

CS4243

Motion Tracking

39

How to handle large displacement?


What if we scale down images?
 Displacements

CS4243

are smaller!

Motion Tracking

40

Image Pyramid
Usually, smoothen
image with Gaussian
filter before scaling
down.

downsample

scaled image

input image
CS4243

Motion Tracking

41

LK Tracking with Image Pyramid

CS4243

Motion Tracking

42



Construct image pyramids.



Apply LK tracker to low-resolution images.



Propagate results to higher-resolution images.



Apply LK tracker to higher-resolution images.

CS4243

Motion Tracking

43

Example

Tracking results are more accurate.

CS4243

Motion Tracking

44

Summary


Efficient algorithm, no explicit search.



Has aperture problem;
track good features only



LK tracker can’t track large displacement.



Use LK + image pyramid for large displacement.

CS4243

Motion Tracking

45

Software


OpenCV supports LK and LK with pyramid.



[Bir] offers LK with Tomasi’s features & pyramid.

CS4243

Motion Tracking

46

Appendix


Calculation of Ix, Iy, It
 Use

finite difference method
 Forward difference
Ix = I(x  1, y, t)  I(x, y, t)
Iy = I(x, y  1, t)  I(x, y, t)
It = I(x, y, t  1)  I(x, y, t)
 Backward difference
Ix = I(x, y, t)  I(x  1, y, t)
Iy = I(x, y, t)  I(x, y  1, t)
It = I(x, y, t)  I(x, y, t  1)

CS4243

Motion Tracking

47

Further Readings


Lucas-Kanade tracking with pyramid:
[BK08] Chapter 10.



Optical flow: [Sze10] Section 8.4.



Hierarchical motion estimation (with image
pyramid): [Sze10] Section 8.1.1.

CS4243

Motion Tracking

48

References


[Bir] S. Birchfield. KLT: An implementation of the Kanade-LucasTomasi feature tracker. http://vision.stanford.edu/~birch/klt/.



[BK08] Bradski and Kaehler. Learning OpenCV: Computer Vision
with the OpenCV Library. O’Reilly, 2008.



[LK81] B. D. Lucas and T. Kanade. An iterative image registration
technique with an application to stereo vision. In Proceedings of 7th
International Joint Conference on Artificial Intelligence, pages 674–
679, 1981.



[ST94] J. Shi and C. Tomasi. Good features to track. In Proceedings
of IEEE Conference on Computer Vision and Pattern Recognition,
pages 593–600, 1994.



[Sze10] R. Szeliski. Computer Vision: Algorithms and Applications.
Springer, 2010.

CS4243

Motion Tracking

49

References


CS4243

[TK91] C. Tomasi and T. Kanade. Detection and tracking of point
features. Technical Report CMU-CS-91-132, School of Computer
Science, Carnegie Mellon University, 1991.

Motion Tracking

50


Slide 33

Leow Wee Kheng
CS4243 Computer Vision and Pattern Recognition

Motion Tracking

CS4243

Motion Tracking

1

Changes are everywhere!

CS4243

Motion Tracking

2

Illumination change

CS4243

Motion Tracking

3

Shape change

CS4243

Motion Tracking

4

Object motion

CS4243

Motion Tracking

5

Camera motion

CS4243

Motion Tracking

6

Object & camera motion

CS4243

Motion Tracking

7

Everything changes!

CS4243

Motion Tracking

8

Motion analysis is tough in general!

We focus on object / camera motion.

CS4243

Motion Tracking

9

Change Detection


Detects any change in two video frames.



Straightforward method:
 Compute

difference between corresponding pixels:



: intensity / colour at (x, y) in frame t.
> threshold, has large difference.

 If

CS4243

Motion Tracking

10

Any difference?

No

CS4243

Motion Tracking

11

Any difference?

Yes,
illumination
change

CS4243

Motion Tracking

12

Any difference?

Yes,
position change

CS4243

Motion Tracking

13

Change Detection


Can detect
 Illumination

change
 Position change
 Illumination and position change


But, cannot distinguish between them.



Need to detect and measure position change.

CS4243

Motion Tracking

14

Motion Tracking


Two approaches
 Feature-based
 Intensity

CS4243

gradient-based

Motion Tracking

15

Feature-based Motion Tracking


Look for distinct features that change positions.

 Eagle’s

wing tips change positions.
 Tree tops don’t change positions.
CS4243

Motion Tracking

16

Basic Ideas
1.

Look for distinct features in current frame.

2.

For each feature


Search for matching feature within neighbourhood
in next frame.
 Difference in positions  displacement.
 Velocity = displacement / time difference.

CS4243

Motion Tracking

17

Basic Ideas
displacement

feature area

search area

CS4243

Motion Tracking

18

What feature to use?


Harris corner



Tomasi’s feature



Feature descriptors: SIFT, SURF, GLOH, etc.



Others

CS4243

Motion Tracking

19

Summary


Simple algorithm.



Can be slow if search area is large.



Can constrain search area with prior knowledge.

CS4243

Motion Tracking

20

Gradient-based Motion Tracking


Two basic assumptions
 Intensity

changes smoothly with position.
 Intensity of object doesn’t change over time.


Suppose an object is in motion.
 Change

position (dx, dy) over time dt.
 Then, from 2nd assumption:
 Apply

CS4243

Taylor’s series expansion:

Motion Tracking

21

 Omit

higher order terms and divide by dt

 Denote

 Then,

 u,

v are unknown.
 2 unknowns, 1 equation: can’t solve!

CS4243

Motion Tracking

22

Any difference in motion?

CS4243

Motion Tracking

23

Any difference in motion?

CS4243

Motion Tracking

24

Any difference in motion?

CS4243

Motion Tracking

25

Any difference in motion?

CS4243

Motion Tracking

26

Any difference in motion?

CS4243

Motion Tracking

27

Any difference in motion?

CS4243

Motion Tracking

28

Aperture Problem


Homogeneous region

 Ix

= Iy = It = 0.
 No change in local region.
 Cannot detect motion.

CS4243

Motion Tracking

29

Aperture Problem


Edge

 Ix

and Iy are zero along edge
 Cannot measure motion tangential to edge
 Ix and Iy are non-zero normal to edge
 Can measure motion normal to edge
 So, cannot measure actual motion

CS4243

Motion Tracking

30

Aperture Problem


Corner

 Ix

and Iy are non-zero in two perpendicular directions
 2 unknowns, 2 equations
 Can measure actual motion

CS4243

Motion Tracking

31

Lucas-Kanade Method


Consider two consecutive image frames I and J:
I

J
x+d
d
x

x

moves from x = (x, y)T to x + d.
 d = (u, v) T
 Object

CS4243

Motion Tracking

32



So,



Or



Due to noise, there’s an error at position x:



Sum error over small window W at position x:

Similar to
template matching
CS4243

weight
Motion Tracking

33



If E is small, patterns in I and J match well.



So, find d that minimises E:
E / d = 0, compute d that minimises E.
 First, expand I(x – d) by Taylor’s series expansion:
 Set

 Omit

higher order terms:

 Write

in matrix form:

Intensity gradient
CS4243

Motion Tracking

34

 Now,

error E at position x is:

 Now,

differentiate E with respect to d (exercise):

 Setting

E / d = 0 gives

b

the only
unknown

Z

CS4243

Motion Tracking

35

 So,

2



CS4243

we get

unknowns, 2 equations. Can solve for d = (u, v).

What happen to aperture problem?
Did it disappear?
Motion Tracking

36

Lucas-Kanade + Tomasi


Lucas-Kanade algorithm is often used with
Tomasi’s feature
 Apply

Tomasi’s method to detect good features.
 Apply LK method to compute d for each pixel.
 Accept d only for good features.

CS4243

Motion Tracking

37

Example

Can you spot tracking errors?

CS4243

Motion Tracking

38

Constraints


Math of LK tracker assumes d is small.



In implementation, W is also small.



LK tracker is good only for small displacement.

CS4243

Motion Tracking

39

How to handle large displacement?


What if we scale down images?
 Displacements

CS4243

are smaller!

Motion Tracking

40

Image Pyramid
Usually, smoothen
image with Gaussian
filter before scaling
down.

downsample

scaled image

input image
CS4243

Motion Tracking

41

LK Tracking with Image Pyramid

CS4243

Motion Tracking

42



Construct image pyramids.



Apply LK tracker to low-resolution images.



Propagate results to higher-resolution images.



Apply LK tracker to higher-resolution images.

CS4243

Motion Tracking

43

Example

Tracking results are more accurate.

CS4243

Motion Tracking

44

Summary


Efficient algorithm, no explicit search.



Has aperture problem;
track good features only



LK tracker can’t track large displacement.



Use LK + image pyramid for large displacement.

CS4243

Motion Tracking

45

Software


OpenCV supports LK and LK with pyramid.



[Bir] offers LK with Tomasi’s features & pyramid.

CS4243

Motion Tracking

46

Appendix


Calculation of Ix, Iy, It
 Use

finite difference method
 Forward difference
Ix = I(x  1, y, t)  I(x, y, t)
Iy = I(x, y  1, t)  I(x, y, t)
It = I(x, y, t  1)  I(x, y, t)
 Backward difference
Ix = I(x, y, t)  I(x  1, y, t)
Iy = I(x, y, t)  I(x, y  1, t)
It = I(x, y, t)  I(x, y, t  1)

CS4243

Motion Tracking

47

Further Readings


Lucas-Kanade tracking with pyramid:
[BK08] Chapter 10.



Optical flow: [Sze10] Section 8.4.



Hierarchical motion estimation (with image
pyramid): [Sze10] Section 8.1.1.

CS4243

Motion Tracking

48

References


[Bir] S. Birchfield. KLT: An implementation of the Kanade-LucasTomasi feature tracker. http://vision.stanford.edu/~birch/klt/.



[BK08] Bradski and Kaehler. Learning OpenCV: Computer Vision
with the OpenCV Library. O’Reilly, 2008.



[LK81] B. D. Lucas and T. Kanade. An iterative image registration
technique with an application to stereo vision. In Proceedings of 7th
International Joint Conference on Artificial Intelligence, pages 674–
679, 1981.



[ST94] J. Shi and C. Tomasi. Good features to track. In Proceedings
of IEEE Conference on Computer Vision and Pattern Recognition,
pages 593–600, 1994.



[Sze10] R. Szeliski. Computer Vision: Algorithms and Applications.
Springer, 2010.

CS4243

Motion Tracking

49

References


CS4243

[TK91] C. Tomasi and T. Kanade. Detection and tracking of point
features. Technical Report CMU-CS-91-132, School of Computer
Science, Carnegie Mellon University, 1991.

Motion Tracking

50


Slide 34

Leow Wee Kheng
CS4243 Computer Vision and Pattern Recognition

Motion Tracking

CS4243

Motion Tracking

1

Changes are everywhere!

CS4243

Motion Tracking

2

Illumination change

CS4243

Motion Tracking

3

Shape change

CS4243

Motion Tracking

4

Object motion

CS4243

Motion Tracking

5

Camera motion

CS4243

Motion Tracking

6

Object & camera motion

CS4243

Motion Tracking

7

Everything changes!

CS4243

Motion Tracking

8

Motion analysis is tough in general!

We focus on object / camera motion.

CS4243

Motion Tracking

9

Change Detection


Detects any change in two video frames.



Straightforward method:
 Compute

difference between corresponding pixels:



: intensity / colour at (x, y) in frame t.
> threshold, has large difference.

 If

CS4243

Motion Tracking

10

Any difference?

No

CS4243

Motion Tracking

11

Any difference?

Yes,
illumination
change

CS4243

Motion Tracking

12

Any difference?

Yes,
position change

CS4243

Motion Tracking

13

Change Detection


Can detect
 Illumination

change
 Position change
 Illumination and position change


But, cannot distinguish between them.



Need to detect and measure position change.

CS4243

Motion Tracking

14

Motion Tracking


Two approaches
 Feature-based
 Intensity

CS4243

gradient-based

Motion Tracking

15

Feature-based Motion Tracking


Look for distinct features that change positions.

 Eagle’s

wing tips change positions.
 Tree tops don’t change positions.
CS4243

Motion Tracking

16

Basic Ideas
1.

Look for distinct features in current frame.

2.

For each feature


Search for matching feature within neighbourhood
in next frame.
 Difference in positions  displacement.
 Velocity = displacement / time difference.

CS4243

Motion Tracking

17

Basic Ideas
displacement

feature area

search area

CS4243

Motion Tracking

18

What feature to use?


Harris corner



Tomasi’s feature



Feature descriptors: SIFT, SURF, GLOH, etc.



Others

CS4243

Motion Tracking

19

Summary


Simple algorithm.



Can be slow if search area is large.



Can constrain search area with prior knowledge.

CS4243

Motion Tracking

20

Gradient-based Motion Tracking


Two basic assumptions
 Intensity

changes smoothly with position.
 Intensity of object doesn’t change over time.


Suppose an object is in motion.
 Change

position (dx, dy) over time dt.
 Then, from 2nd assumption:
 Apply

CS4243

Taylor’s series expansion:

Motion Tracking

21

 Omit

higher order terms and divide by dt

 Denote

 Then,

 u,

v are unknown.
 2 unknowns, 1 equation: can’t solve!

CS4243

Motion Tracking

22

Any difference in motion?

CS4243

Motion Tracking

23

Any difference in motion?

CS4243

Motion Tracking

24

Any difference in motion?

CS4243

Motion Tracking

25

Any difference in motion?

CS4243

Motion Tracking

26

Any difference in motion?

CS4243

Motion Tracking

27

Any difference in motion?

CS4243

Motion Tracking

28

Aperture Problem


Homogeneous region

 Ix

= Iy = It = 0.
 No change in local region.
 Cannot detect motion.

CS4243

Motion Tracking

29

Aperture Problem


Edge

 Ix

and Iy are zero along edge
 Cannot measure motion tangential to edge
 Ix and Iy are non-zero normal to edge
 Can measure motion normal to edge
 So, cannot measure actual motion

CS4243

Motion Tracking

30

Aperture Problem


Corner

 Ix

and Iy are non-zero in two perpendicular directions
 2 unknowns, 2 equations
 Can measure actual motion

CS4243

Motion Tracking

31

Lucas-Kanade Method


Consider two consecutive image frames I and J:
I

J
x+d
d
x

x

moves from x = (x, y)T to x + d.
 d = (u, v) T
 Object

CS4243

Motion Tracking

32



So,



Or



Due to noise, there’s an error at position x:



Sum error over small window W at position x:

Similar to
template matching
CS4243

weight
Motion Tracking

33



If E is small, patterns in I and J match well.



So, find d that minimises E:
E / d = 0, compute d that minimises E.
 First, expand I(x – d) by Taylor’s series expansion:
 Set

 Omit

higher order terms:

 Write

in matrix form:

Intensity gradient
CS4243

Motion Tracking

34

 Now,

error E at position x is:

 Now,

differentiate E with respect to d (exercise):

 Setting

E / d = 0 gives

b

the only
unknown

Z

CS4243

Motion Tracking

35

 So,

2



CS4243

we get

unknowns, 2 equations. Can solve for d = (u, v).

What happen to aperture problem?
Did it disappear?
Motion Tracking

36

Lucas-Kanade + Tomasi


Lucas-Kanade algorithm is often used with
Tomasi’s feature
 Apply

Tomasi’s method to detect good features.
 Apply LK method to compute d for each pixel.
 Accept d only for good features.

CS4243

Motion Tracking

37

Example

Can you spot tracking errors?

CS4243

Motion Tracking

38

Constraints


Math of LK tracker assumes d is small.



In implementation, W is also small.



LK tracker is good only for small displacement.

CS4243

Motion Tracking

39

How to handle large displacement?


What if we scale down images?
 Displacements

CS4243

are smaller!

Motion Tracking

40

Image Pyramid
Usually, smoothen
image with Gaussian
filter before scaling
down.

downsample

scaled image

input image
CS4243

Motion Tracking

41

LK Tracking with Image Pyramid

CS4243

Motion Tracking

42



Construct image pyramids.



Apply LK tracker to low-resolution images.



Propagate results to higher-resolution images.



Apply LK tracker to higher-resolution images.

CS4243

Motion Tracking

43

Example

Tracking results are more accurate.

CS4243

Motion Tracking

44

Summary


Efficient algorithm, no explicit search.



Has aperture problem;
track good features only



LK tracker can’t track large displacement.



Use LK + image pyramid for large displacement.

CS4243

Motion Tracking

45

Software


OpenCV supports LK and LK with pyramid.



[Bir] offers LK with Tomasi’s features & pyramid.

CS4243

Motion Tracking

46

Appendix


Calculation of Ix, Iy, It
 Use

finite difference method
 Forward difference
Ix = I(x  1, y, t)  I(x, y, t)
Iy = I(x, y  1, t)  I(x, y, t)
It = I(x, y, t  1)  I(x, y, t)
 Backward difference
Ix = I(x, y, t)  I(x  1, y, t)
Iy = I(x, y, t)  I(x, y  1, t)
It = I(x, y, t)  I(x, y, t  1)

CS4243

Motion Tracking

47

Further Readings


Lucas-Kanade tracking with pyramid:
[BK08] Chapter 10.



Optical flow: [Sze10] Section 8.4.



Hierarchical motion estimation (with image
pyramid): [Sze10] Section 8.1.1.

CS4243

Motion Tracking

48

References


[Bir] S. Birchfield. KLT: An implementation of the Kanade-LucasTomasi feature tracker. http://vision.stanford.edu/~birch/klt/.



[BK08] Bradski and Kaehler. Learning OpenCV: Computer Vision
with the OpenCV Library. O’Reilly, 2008.



[LK81] B. D. Lucas and T. Kanade. An iterative image registration
technique with an application to stereo vision. In Proceedings of 7th
International Joint Conference on Artificial Intelligence, pages 674–
679, 1981.



[ST94] J. Shi and C. Tomasi. Good features to track. In Proceedings
of IEEE Conference on Computer Vision and Pattern Recognition,
pages 593–600, 1994.



[Sze10] R. Szeliski. Computer Vision: Algorithms and Applications.
Springer, 2010.

CS4243

Motion Tracking

49

References


CS4243

[TK91] C. Tomasi and T. Kanade. Detection and tracking of point
features. Technical Report CMU-CS-91-132, School of Computer
Science, Carnegie Mellon University, 1991.

Motion Tracking

50


Slide 35

Leow Wee Kheng
CS4243 Computer Vision and Pattern Recognition

Motion Tracking

CS4243

Motion Tracking

1

Changes are everywhere!

CS4243

Motion Tracking

2

Illumination change

CS4243

Motion Tracking

3

Shape change

CS4243

Motion Tracking

4

Object motion

CS4243

Motion Tracking

5

Camera motion

CS4243

Motion Tracking

6

Object & camera motion

CS4243

Motion Tracking

7

Everything changes!

CS4243

Motion Tracking

8

Motion analysis is tough in general!

We focus on object / camera motion.

CS4243

Motion Tracking

9

Change Detection


Detects any change in two video frames.



Straightforward method:
 Compute

difference between corresponding pixels:



: intensity / colour at (x, y) in frame t.
> threshold, has large difference.

 If

CS4243

Motion Tracking

10

Any difference?

No

CS4243

Motion Tracking

11

Any difference?

Yes,
illumination
change

CS4243

Motion Tracking

12

Any difference?

Yes,
position change

CS4243

Motion Tracking

13

Change Detection


Can detect
 Illumination

change
 Position change
 Illumination and position change


But, cannot distinguish between them.



Need to detect and measure position change.

CS4243

Motion Tracking

14

Motion Tracking


Two approaches
 Feature-based
 Intensity

CS4243

gradient-based

Motion Tracking

15

Feature-based Motion Tracking


Look for distinct features that change positions.

 Eagle’s

wing tips change positions.
 Tree tops don’t change positions.
CS4243

Motion Tracking

16

Basic Ideas
1.

Look for distinct features in current frame.

2.

For each feature


Search for matching feature within neighbourhood
in next frame.
 Difference in positions  displacement.
 Velocity = displacement / time difference.

CS4243

Motion Tracking

17

Basic Ideas
displacement

feature area

search area

CS4243

Motion Tracking

18

What feature to use?


Harris corner



Tomasi’s feature



Feature descriptors: SIFT, SURF, GLOH, etc.



Others

CS4243

Motion Tracking

19

Summary


Simple algorithm.



Can be slow if search area is large.



Can constrain search area with prior knowledge.

CS4243

Motion Tracking

20

Gradient-based Motion Tracking


Two basic assumptions
 Intensity

changes smoothly with position.
 Intensity of object doesn’t change over time.


Suppose an object is in motion.
 Change

position (dx, dy) over time dt.
 Then, from 2nd assumption:
 Apply

CS4243

Taylor’s series expansion:

Motion Tracking

21

 Omit

higher order terms and divide by dt

 Denote

 Then,

 u,

v are unknown.
 2 unknowns, 1 equation: can’t solve!

CS4243

Motion Tracking

22

Any difference in motion?

CS4243

Motion Tracking

23

Any difference in motion?

CS4243

Motion Tracking

24

Any difference in motion?

CS4243

Motion Tracking

25

Any difference in motion?

CS4243

Motion Tracking

26

Any difference in motion?

CS4243

Motion Tracking

27

Any difference in motion?

CS4243

Motion Tracking

28

Aperture Problem


Homogeneous region

 Ix

= Iy = It = 0.
 No change in local region.
 Cannot detect motion.

CS4243

Motion Tracking

29

Aperture Problem


Edge

 Ix

and Iy are zero along edge
 Cannot measure motion tangential to edge
 Ix and Iy are non-zero normal to edge
 Can measure motion normal to edge
 So, cannot measure actual motion

CS4243

Motion Tracking

30

Aperture Problem


Corner

 Ix

and Iy are non-zero in two perpendicular directions
 2 unknowns, 2 equations
 Can measure actual motion

CS4243

Motion Tracking

31

Lucas-Kanade Method


Consider two consecutive image frames I and J:
I

J
x+d
d
x

x

moves from x = (x, y)T to x + d.
 d = (u, v) T
 Object

CS4243

Motion Tracking

32



So,



Or



Due to noise, there’s an error at position x:



Sum error over small window W at position x:

Similar to
template matching
CS4243

weight
Motion Tracking

33



If E is small, patterns in I and J match well.



So, find d that minimises E:
E / d = 0, compute d that minimises E.
 First, expand I(x – d) by Taylor’s series expansion:
 Set

 Omit

higher order terms:

 Write

in matrix form:

Intensity gradient
CS4243

Motion Tracking

34

 Now,

error E at position x is:

 Now,

differentiate E with respect to d (exercise):

 Setting

E / d = 0 gives

b

the only
unknown

Z

CS4243

Motion Tracking

35

 So,

2



CS4243

we get

unknowns, 2 equations. Can solve for d = (u, v).

What happen to aperture problem?
Did it disappear?
Motion Tracking

36

Lucas-Kanade + Tomasi


Lucas-Kanade algorithm is often used with
Tomasi’s feature
 Apply

Tomasi’s method to detect good features.
 Apply LK method to compute d for each pixel.
 Accept d only for good features.

CS4243

Motion Tracking

37

Example

Can you spot tracking errors?

CS4243

Motion Tracking

38

Constraints


Math of LK tracker assumes d is small.



In implementation, W is also small.



LK tracker is good only for small displacement.

CS4243

Motion Tracking

39

How to handle large displacement?


What if we scale down images?
 Displacements

CS4243

are smaller!

Motion Tracking

40

Image Pyramid
Usually, smoothen
image with Gaussian
filter before scaling
down.

downsample

scaled image

input image
CS4243

Motion Tracking

41

LK Tracking with Image Pyramid

CS4243

Motion Tracking

42



Construct image pyramids.



Apply LK tracker to low-resolution images.



Propagate results to higher-resolution images.



Apply LK tracker to higher-resolution images.

CS4243

Motion Tracking

43

Example

Tracking results are more accurate.

CS4243

Motion Tracking

44

Summary


Efficient algorithm, no explicit search.



Has aperture problem;
track good features only



LK tracker can’t track large displacement.



Use LK + image pyramid for large displacement.

CS4243

Motion Tracking

45

Software


OpenCV supports LK and LK with pyramid.



[Bir] offers LK with Tomasi’s features & pyramid.

CS4243

Motion Tracking

46

Appendix


Calculation of Ix, Iy, It
 Use

finite difference method
 Forward difference
Ix = I(x  1, y, t)  I(x, y, t)
Iy = I(x, y  1, t)  I(x, y, t)
It = I(x, y, t  1)  I(x, y, t)
 Backward difference
Ix = I(x, y, t)  I(x  1, y, t)
Iy = I(x, y, t)  I(x, y  1, t)
It = I(x, y, t)  I(x, y, t  1)

CS4243

Motion Tracking

47

Further Readings


Lucas-Kanade tracking with pyramid:
[BK08] Chapter 10.



Optical flow: [Sze10] Section 8.4.



Hierarchical motion estimation (with image
pyramid): [Sze10] Section 8.1.1.

CS4243

Motion Tracking

48

References


[Bir] S. Birchfield. KLT: An implementation of the Kanade-LucasTomasi feature tracker. http://vision.stanford.edu/~birch/klt/.



[BK08] Bradski and Kaehler. Learning OpenCV: Computer Vision
with the OpenCV Library. O’Reilly, 2008.



[LK81] B. D. Lucas and T. Kanade. An iterative image registration
technique with an application to stereo vision. In Proceedings of 7th
International Joint Conference on Artificial Intelligence, pages 674–
679, 1981.



[ST94] J. Shi and C. Tomasi. Good features to track. In Proceedings
of IEEE Conference on Computer Vision and Pattern Recognition,
pages 593–600, 1994.



[Sze10] R. Szeliski. Computer Vision: Algorithms and Applications.
Springer, 2010.

CS4243

Motion Tracking

49

References


CS4243

[TK91] C. Tomasi and T. Kanade. Detection and tracking of point
features. Technical Report CMU-CS-91-132, School of Computer
Science, Carnegie Mellon University, 1991.

Motion Tracking

50


Slide 36

Leow Wee Kheng
CS4243 Computer Vision and Pattern Recognition

Motion Tracking

CS4243

Motion Tracking

1

Changes are everywhere!

CS4243

Motion Tracking

2

Illumination change

CS4243

Motion Tracking

3

Shape change

CS4243

Motion Tracking

4

Object motion

CS4243

Motion Tracking

5

Camera motion

CS4243

Motion Tracking

6

Object & camera motion

CS4243

Motion Tracking

7

Everything changes!

CS4243

Motion Tracking

8

Motion analysis is tough in general!

We focus on object / camera motion.

CS4243

Motion Tracking

9

Change Detection


Detects any change in two video frames.



Straightforward method:
 Compute

difference between corresponding pixels:



: intensity / colour at (x, y) in frame t.
> threshold, has large difference.

 If

CS4243

Motion Tracking

10

Any difference?

No

CS4243

Motion Tracking

11

Any difference?

Yes,
illumination
change

CS4243

Motion Tracking

12

Any difference?

Yes,
position change

CS4243

Motion Tracking

13

Change Detection


Can detect
 Illumination

change
 Position change
 Illumination and position change


But, cannot distinguish between them.



Need to detect and measure position change.

CS4243

Motion Tracking

14

Motion Tracking


Two approaches
 Feature-based
 Intensity

CS4243

gradient-based

Motion Tracking

15

Feature-based Motion Tracking


Look for distinct features that change positions.

 Eagle’s

wing tips change positions.
 Tree tops don’t change positions.
CS4243

Motion Tracking

16

Basic Ideas
1.

Look for distinct features in current frame.

2.

For each feature


Search for matching feature within neighbourhood
in next frame.
 Difference in positions  displacement.
 Velocity = displacement / time difference.

CS4243

Motion Tracking

17

Basic Ideas
displacement

feature area

search area

CS4243

Motion Tracking

18

What feature to use?


Harris corner



Tomasi’s feature



Feature descriptors: SIFT, SURF, GLOH, etc.



Others

CS4243

Motion Tracking

19

Summary


Simple algorithm.



Can be slow if search area is large.



Can constrain search area with prior knowledge.

CS4243

Motion Tracking

20

Gradient-based Motion Tracking


Two basic assumptions
 Intensity

changes smoothly with position.
 Intensity of object doesn’t change over time.


Suppose an object is in motion.
 Change

position (dx, dy) over time dt.
 Then, from 2nd assumption:
 Apply

CS4243

Taylor’s series expansion:

Motion Tracking

21

 Omit

higher order terms and divide by dt

 Denote

 Then,

 u,

v are unknown.
 2 unknowns, 1 equation: can’t solve!

CS4243

Motion Tracking

22

Any difference in motion?

CS4243

Motion Tracking

23

Any difference in motion?

CS4243

Motion Tracking

24

Any difference in motion?

CS4243

Motion Tracking

25

Any difference in motion?

CS4243

Motion Tracking

26

Any difference in motion?

CS4243

Motion Tracking

27

Any difference in motion?

CS4243

Motion Tracking

28

Aperture Problem


Homogeneous region

 Ix

= Iy = It = 0.
 No change in local region.
 Cannot detect motion.

CS4243

Motion Tracking

29

Aperture Problem


Edge

 Ix

and Iy are zero along edge
 Cannot measure motion tangential to edge
 Ix and Iy are non-zero normal to edge
 Can measure motion normal to edge
 So, cannot measure actual motion

CS4243

Motion Tracking

30

Aperture Problem


Corner

 Ix

and Iy are non-zero in two perpendicular directions
 2 unknowns, 2 equations
 Can measure actual motion

CS4243

Motion Tracking

31

Lucas-Kanade Method


Consider two consecutive image frames I and J:
I

J
x+d
d
x

x

moves from x = (x, y)T to x + d.
 d = (u, v) T
 Object

CS4243

Motion Tracking

32



So,



Or



Due to noise, there’s an error at position x:



Sum error over small window W at position x:

Similar to
template matching
CS4243

weight
Motion Tracking

33



If E is small, patterns in I and J match well.



So, find d that minimises E:
E / d = 0, compute d that minimises E.
 First, expand I(x – d) by Taylor’s series expansion:
 Set

 Omit

higher order terms:

 Write

in matrix form:

Intensity gradient
CS4243

Motion Tracking

34

 Now,

error E at position x is:

 Now,

differentiate E with respect to d (exercise):

 Setting

E / d = 0 gives

b

the only
unknown

Z

CS4243

Motion Tracking

35

 So,

2



CS4243

we get

unknowns, 2 equations. Can solve for d = (u, v).

What happen to aperture problem?
Did it disappear?
Motion Tracking

36

Lucas-Kanade + Tomasi


Lucas-Kanade algorithm is often used with
Tomasi’s feature
 Apply

Tomasi’s method to detect good features.
 Apply LK method to compute d for each pixel.
 Accept d only for good features.

CS4243

Motion Tracking

37

Example

Can you spot tracking errors?

CS4243

Motion Tracking

38

Constraints


Math of LK tracker assumes d is small.



In implementation, W is also small.



LK tracker is good only for small displacement.

CS4243

Motion Tracking

39

How to handle large displacement?


What if we scale down images?
 Displacements

CS4243

are smaller!

Motion Tracking

40

Image Pyramid
Usually, smoothen
image with Gaussian
filter before scaling
down.

downsample

scaled image

input image
CS4243

Motion Tracking

41

LK Tracking with Image Pyramid

CS4243

Motion Tracking

42



Construct image pyramids.



Apply LK tracker to low-resolution images.



Propagate results to higher-resolution images.



Apply LK tracker to higher-resolution images.

CS4243

Motion Tracking

43

Example

Tracking results are more accurate.

CS4243

Motion Tracking

44

Summary


Efficient algorithm, no explicit search.



Has aperture problem;
track good features only



LK tracker can’t track large displacement.



Use LK + image pyramid for large displacement.

CS4243

Motion Tracking

45

Software


OpenCV supports LK and LK with pyramid.



[Bir] offers LK with Tomasi’s features & pyramid.

CS4243

Motion Tracking

46

Appendix


Calculation of Ix, Iy, It
 Use

finite difference method
 Forward difference
Ix = I(x  1, y, t)  I(x, y, t)
Iy = I(x, y  1, t)  I(x, y, t)
It = I(x, y, t  1)  I(x, y, t)
 Backward difference
Ix = I(x, y, t)  I(x  1, y, t)
Iy = I(x, y, t)  I(x, y  1, t)
It = I(x, y, t)  I(x, y, t  1)

CS4243

Motion Tracking

47

Further Readings


Lucas-Kanade tracking with pyramid:
[BK08] Chapter 10.



Optical flow: [Sze10] Section 8.4.



Hierarchical motion estimation (with image
pyramid): [Sze10] Section 8.1.1.

CS4243

Motion Tracking

48

References


[Bir] S. Birchfield. KLT: An implementation of the Kanade-LucasTomasi feature tracker. http://vision.stanford.edu/~birch/klt/.



[BK08] Bradski and Kaehler. Learning OpenCV: Computer Vision
with the OpenCV Library. O’Reilly, 2008.



[LK81] B. D. Lucas and T. Kanade. An iterative image registration
technique with an application to stereo vision. In Proceedings of 7th
International Joint Conference on Artificial Intelligence, pages 674–
679, 1981.



[ST94] J. Shi and C. Tomasi. Good features to track. In Proceedings
of IEEE Conference on Computer Vision and Pattern Recognition,
pages 593–600, 1994.



[Sze10] R. Szeliski. Computer Vision: Algorithms and Applications.
Springer, 2010.

CS4243

Motion Tracking

49

References


CS4243

[TK91] C. Tomasi and T. Kanade. Detection and tracking of point
features. Technical Report CMU-CS-91-132, School of Computer
Science, Carnegie Mellon University, 1991.

Motion Tracking

50


Slide 37

Leow Wee Kheng
CS4243 Computer Vision and Pattern Recognition

Motion Tracking

CS4243

Motion Tracking

1

Changes are everywhere!

CS4243

Motion Tracking

2

Illumination change

CS4243

Motion Tracking

3

Shape change

CS4243

Motion Tracking

4

Object motion

CS4243

Motion Tracking

5

Camera motion

CS4243

Motion Tracking

6

Object & camera motion

CS4243

Motion Tracking

7

Everything changes!

CS4243

Motion Tracking

8

Motion analysis is tough in general!

We focus on object / camera motion.

CS4243

Motion Tracking

9

Change Detection


Detects any change in two video frames.



Straightforward method:
 Compute

difference between corresponding pixels:



: intensity / colour at (x, y) in frame t.
> threshold, has large difference.

 If

CS4243

Motion Tracking

10

Any difference?

No

CS4243

Motion Tracking

11

Any difference?

Yes,
illumination
change

CS4243

Motion Tracking

12

Any difference?

Yes,
position change

CS4243

Motion Tracking

13

Change Detection


Can detect
 Illumination

change
 Position change
 Illumination and position change


But, cannot distinguish between them.



Need to detect and measure position change.

CS4243

Motion Tracking

14

Motion Tracking


Two approaches
 Feature-based
 Intensity

CS4243

gradient-based

Motion Tracking

15

Feature-based Motion Tracking


Look for distinct features that change positions.

 Eagle’s

wing tips change positions.
 Tree tops don’t change positions.
CS4243

Motion Tracking

16

Basic Ideas
1.

Look for distinct features in current frame.

2.

For each feature


Search for matching feature within neighbourhood
in next frame.
 Difference in positions  displacement.
 Velocity = displacement / time difference.

CS4243

Motion Tracking

17

Basic Ideas
displacement

feature area

search area

CS4243

Motion Tracking

18

What feature to use?


Harris corner



Tomasi’s feature



Feature descriptors: SIFT, SURF, GLOH, etc.



Others

CS4243

Motion Tracking

19

Summary


Simple algorithm.



Can be slow if search area is large.



Can constrain search area with prior knowledge.

CS4243

Motion Tracking

20

Gradient-based Motion Tracking


Two basic assumptions
 Intensity

changes smoothly with position.
 Intensity of object doesn’t change over time.


Suppose an object is in motion.
 Change

position (dx, dy) over time dt.
 Then, from 2nd assumption:
 Apply

CS4243

Taylor’s series expansion:

Motion Tracking

21

 Omit

higher order terms and divide by dt

 Denote

 Then,

 u,

v are unknown.
 2 unknowns, 1 equation: can’t solve!

CS4243

Motion Tracking

22

Any difference in motion?

CS4243

Motion Tracking

23

Any difference in motion?

CS4243

Motion Tracking

24

Any difference in motion?

CS4243

Motion Tracking

25

Any difference in motion?

CS4243

Motion Tracking

26

Any difference in motion?

CS4243

Motion Tracking

27

Any difference in motion?

CS4243

Motion Tracking

28

Aperture Problem


Homogeneous region

 Ix

= Iy = It = 0.
 No change in local region.
 Cannot detect motion.

CS4243

Motion Tracking

29

Aperture Problem


Edge

 Ix

and Iy are zero along edge
 Cannot measure motion tangential to edge
 Ix and Iy are non-zero normal to edge
 Can measure motion normal to edge
 So, cannot measure actual motion

CS4243

Motion Tracking

30

Aperture Problem


Corner

 Ix

and Iy are non-zero in two perpendicular directions
 2 unknowns, 2 equations
 Can measure actual motion

CS4243

Motion Tracking

31

Lucas-Kanade Method


Consider two consecutive image frames I and J:
I

J
x+d
d
x

x

moves from x = (x, y)T to x + d.
 d = (u, v) T
 Object

CS4243

Motion Tracking

32



So,



Or



Due to noise, there’s an error at position x:



Sum error over small window W at position x:

Similar to
template matching
CS4243

weight
Motion Tracking

33



If E is small, patterns in I and J match well.



So, find d that minimises E:
E / d = 0, compute d that minimises E.
 First, expand I(x – d) by Taylor’s series expansion:
 Set

 Omit

higher order terms:

 Write

in matrix form:

Intensity gradient
CS4243

Motion Tracking

34

 Now,

error E at position x is:

 Now,

differentiate E with respect to d (exercise):

 Setting

E / d = 0 gives

b

the only
unknown

Z

CS4243

Motion Tracking

35

 So,

2



CS4243

we get

unknowns, 2 equations. Can solve for d = (u, v).

What happen to aperture problem?
Did it disappear?
Motion Tracking

36

Lucas-Kanade + Tomasi


Lucas-Kanade algorithm is often used with
Tomasi’s feature
 Apply

Tomasi’s method to detect good features.
 Apply LK method to compute d for each pixel.
 Accept d only for good features.

CS4243

Motion Tracking

37

Example

Can you spot tracking errors?

CS4243

Motion Tracking

38

Constraints


Math of LK tracker assumes d is small.



In implementation, W is also small.



LK tracker is good only for small displacement.

CS4243

Motion Tracking

39

How to handle large displacement?


What if we scale down images?
 Displacements

CS4243

are smaller!

Motion Tracking

40

Image Pyramid
Usually, smoothen
image with Gaussian
filter before scaling
down.

downsample

scaled image

input image
CS4243

Motion Tracking

41

LK Tracking with Image Pyramid

CS4243

Motion Tracking

42



Construct image pyramids.



Apply LK tracker to low-resolution images.



Propagate results to higher-resolution images.



Apply LK tracker to higher-resolution images.

CS4243

Motion Tracking

43

Example

Tracking results are more accurate.

CS4243

Motion Tracking

44

Summary


Efficient algorithm, no explicit search.



Has aperture problem;
track good features only



LK tracker can’t track large displacement.



Use LK + image pyramid for large displacement.

CS4243

Motion Tracking

45

Software


OpenCV supports LK and LK with pyramid.



[Bir] offers LK with Tomasi’s features & pyramid.

CS4243

Motion Tracking

46

Appendix


Calculation of Ix, Iy, It
 Use

finite difference method
 Forward difference
Ix = I(x  1, y, t)  I(x, y, t)
Iy = I(x, y  1, t)  I(x, y, t)
It = I(x, y, t  1)  I(x, y, t)
 Backward difference
Ix = I(x, y, t)  I(x  1, y, t)
Iy = I(x, y, t)  I(x, y  1, t)
It = I(x, y, t)  I(x, y, t  1)

CS4243

Motion Tracking

47

Further Readings


Lucas-Kanade tracking with pyramid:
[BK08] Chapter 10.



Optical flow: [Sze10] Section 8.4.



Hierarchical motion estimation (with image
pyramid): [Sze10] Section 8.1.1.

CS4243

Motion Tracking

48

References


[Bir] S. Birchfield. KLT: An implementation of the Kanade-LucasTomasi feature tracker. http://vision.stanford.edu/~birch/klt/.



[BK08] Bradski and Kaehler. Learning OpenCV: Computer Vision
with the OpenCV Library. O’Reilly, 2008.



[LK81] B. D. Lucas and T. Kanade. An iterative image registration
technique with an application to stereo vision. In Proceedings of 7th
International Joint Conference on Artificial Intelligence, pages 674–
679, 1981.



[ST94] J. Shi and C. Tomasi. Good features to track. In Proceedings
of IEEE Conference on Computer Vision and Pattern Recognition,
pages 593–600, 1994.



[Sze10] R. Szeliski. Computer Vision: Algorithms and Applications.
Springer, 2010.

CS4243

Motion Tracking

49

References


CS4243

[TK91] C. Tomasi and T. Kanade. Detection and tracking of point
features. Technical Report CMU-CS-91-132, School of Computer
Science, Carnegie Mellon University, 1991.

Motion Tracking

50


Slide 38

Leow Wee Kheng
CS4243 Computer Vision and Pattern Recognition

Motion Tracking

CS4243

Motion Tracking

1

Changes are everywhere!

CS4243

Motion Tracking

2

Illumination change

CS4243

Motion Tracking

3

Shape change

CS4243

Motion Tracking

4

Object motion

CS4243

Motion Tracking

5

Camera motion

CS4243

Motion Tracking

6

Object & camera motion

CS4243

Motion Tracking

7

Everything changes!

CS4243

Motion Tracking

8

Motion analysis is tough in general!

We focus on object / camera motion.

CS4243

Motion Tracking

9

Change Detection


Detects any change in two video frames.



Straightforward method:
 Compute

difference between corresponding pixels:



: intensity / colour at (x, y) in frame t.
> threshold, has large difference.

 If

CS4243

Motion Tracking

10

Any difference?

No

CS4243

Motion Tracking

11

Any difference?

Yes,
illumination
change

CS4243

Motion Tracking

12

Any difference?

Yes,
position change

CS4243

Motion Tracking

13

Change Detection


Can detect
 Illumination

change
 Position change
 Illumination and position change


But, cannot distinguish between them.



Need to detect and measure position change.

CS4243

Motion Tracking

14

Motion Tracking


Two approaches
 Feature-based
 Intensity

CS4243

gradient-based

Motion Tracking

15

Feature-based Motion Tracking


Look for distinct features that change positions.

 Eagle’s

wing tips change positions.
 Tree tops don’t change positions.
CS4243

Motion Tracking

16

Basic Ideas
1.

Look for distinct features in current frame.

2.

For each feature


Search for matching feature within neighbourhood
in next frame.
 Difference in positions  displacement.
 Velocity = displacement / time difference.

CS4243

Motion Tracking

17

Basic Ideas
displacement

feature area

search area

CS4243

Motion Tracking

18

What feature to use?


Harris corner



Tomasi’s feature



Feature descriptors: SIFT, SURF, GLOH, etc.



Others

CS4243

Motion Tracking

19

Summary


Simple algorithm.



Can be slow if search area is large.



Can constrain search area with prior knowledge.

CS4243

Motion Tracking

20

Gradient-based Motion Tracking


Two basic assumptions
 Intensity

changes smoothly with position.
 Intensity of object doesn’t change over time.


Suppose an object is in motion.
 Change

position (dx, dy) over time dt.
 Then, from 2nd assumption:
 Apply

CS4243

Taylor’s series expansion:

Motion Tracking

21

 Omit

higher order terms and divide by dt

 Denote

 Then,

 u,

v are unknown.
 2 unknowns, 1 equation: can’t solve!

CS4243

Motion Tracking

22

Any difference in motion?

CS4243

Motion Tracking

23

Any difference in motion?

CS4243

Motion Tracking

24

Any difference in motion?

CS4243

Motion Tracking

25

Any difference in motion?

CS4243

Motion Tracking

26

Any difference in motion?

CS4243

Motion Tracking

27

Any difference in motion?

CS4243

Motion Tracking

28

Aperture Problem


Homogeneous region

 Ix

= Iy = It = 0.
 No change in local region.
 Cannot detect motion.

CS4243

Motion Tracking

29

Aperture Problem


Edge

 Ix

and Iy are zero along edge
 Cannot measure motion tangential to edge
 Ix and Iy are non-zero normal to edge
 Can measure motion normal to edge
 So, cannot measure actual motion

CS4243

Motion Tracking

30

Aperture Problem


Corner

 Ix

and Iy are non-zero in two perpendicular directions
 2 unknowns, 2 equations
 Can measure actual motion

CS4243

Motion Tracking

31

Lucas-Kanade Method


Consider two consecutive image frames I and J:
I

J
x+d
d
x

x

moves from x = (x, y)T to x + d.
 d = (u, v) T
 Object

CS4243

Motion Tracking

32



So,



Or



Due to noise, there’s an error at position x:



Sum error over small window W at position x:

Similar to
template matching
CS4243

weight
Motion Tracking

33



If E is small, patterns in I and J match well.



So, find d that minimises E:
E / d = 0, compute d that minimises E.
 First, expand I(x – d) by Taylor’s series expansion:
 Set

 Omit

higher order terms:

 Write

in matrix form:

Intensity gradient
CS4243

Motion Tracking

34

 Now,

error E at position x is:

 Now,

differentiate E with respect to d (exercise):

 Setting

E / d = 0 gives

b

the only
unknown

Z

CS4243

Motion Tracking

35

 So,

2



CS4243

we get

unknowns, 2 equations. Can solve for d = (u, v).

What happen to aperture problem?
Did it disappear?
Motion Tracking

36

Lucas-Kanade + Tomasi


Lucas-Kanade algorithm is often used with
Tomasi’s feature
 Apply

Tomasi’s method to detect good features.
 Apply LK method to compute d for each pixel.
 Accept d only for good features.

CS4243

Motion Tracking

37

Example

Can you spot tracking errors?

CS4243

Motion Tracking

38

Constraints


Math of LK tracker assumes d is small.



In implementation, W is also small.



LK tracker is good only for small displacement.

CS4243

Motion Tracking

39

How to handle large displacement?


What if we scale down images?
 Displacements

CS4243

are smaller!

Motion Tracking

40

Image Pyramid
Usually, smoothen
image with Gaussian
filter before scaling
down.

downsample

scaled image

input image
CS4243

Motion Tracking

41

LK Tracking with Image Pyramid

CS4243

Motion Tracking

42



Construct image pyramids.



Apply LK tracker to low-resolution images.



Propagate results to higher-resolution images.



Apply LK tracker to higher-resolution images.

CS4243

Motion Tracking

43

Example

Tracking results are more accurate.

CS4243

Motion Tracking

44

Summary


Efficient algorithm, no explicit search.



Has aperture problem;
track good features only



LK tracker can’t track large displacement.



Use LK + image pyramid for large displacement.

CS4243

Motion Tracking

45

Software


OpenCV supports LK and LK with pyramid.



[Bir] offers LK with Tomasi’s features & pyramid.

CS4243

Motion Tracking

46

Appendix


Calculation of Ix, Iy, It
 Use

finite difference method
 Forward difference
Ix = I(x  1, y, t)  I(x, y, t)
Iy = I(x, y  1, t)  I(x, y, t)
It = I(x, y, t  1)  I(x, y, t)
 Backward difference
Ix = I(x, y, t)  I(x  1, y, t)
Iy = I(x, y, t)  I(x, y  1, t)
It = I(x, y, t)  I(x, y, t  1)

CS4243

Motion Tracking

47

Further Readings


Lucas-Kanade tracking with pyramid:
[BK08] Chapter 10.



Optical flow: [Sze10] Section 8.4.



Hierarchical motion estimation (with image
pyramid): [Sze10] Section 8.1.1.

CS4243

Motion Tracking

48

References


[Bir] S. Birchfield. KLT: An implementation of the Kanade-LucasTomasi feature tracker. http://vision.stanford.edu/~birch/klt/.



[BK08] Bradski and Kaehler. Learning OpenCV: Computer Vision
with the OpenCV Library. O’Reilly, 2008.



[LK81] B. D. Lucas and T. Kanade. An iterative image registration
technique with an application to stereo vision. In Proceedings of 7th
International Joint Conference on Artificial Intelligence, pages 674–
679, 1981.



[ST94] J. Shi and C. Tomasi. Good features to track. In Proceedings
of IEEE Conference on Computer Vision and Pattern Recognition,
pages 593–600, 1994.



[Sze10] R. Szeliski. Computer Vision: Algorithms and Applications.
Springer, 2010.

CS4243

Motion Tracking

49

References


CS4243

[TK91] C. Tomasi and T. Kanade. Detection and tracking of point
features. Technical Report CMU-CS-91-132, School of Computer
Science, Carnegie Mellon University, 1991.

Motion Tracking

50


Slide 39

Leow Wee Kheng
CS4243 Computer Vision and Pattern Recognition

Motion Tracking

CS4243

Motion Tracking

1

Changes are everywhere!

CS4243

Motion Tracking

2

Illumination change

CS4243

Motion Tracking

3

Shape change

CS4243

Motion Tracking

4

Object motion

CS4243

Motion Tracking

5

Camera motion

CS4243

Motion Tracking

6

Object & camera motion

CS4243

Motion Tracking

7

Everything changes!

CS4243

Motion Tracking

8

Motion analysis is tough in general!

We focus on object / camera motion.

CS4243

Motion Tracking

9

Change Detection


Detects any change in two video frames.



Straightforward method:
 Compute

difference between corresponding pixels:



: intensity / colour at (x, y) in frame t.
> threshold, has large difference.

 If

CS4243

Motion Tracking

10

Any difference?

No

CS4243

Motion Tracking

11

Any difference?

Yes,
illumination
change

CS4243

Motion Tracking

12

Any difference?

Yes,
position change

CS4243

Motion Tracking

13

Change Detection


Can detect
 Illumination

change
 Position change
 Illumination and position change


But, cannot distinguish between them.



Need to detect and measure position change.

CS4243

Motion Tracking

14

Motion Tracking


Two approaches
 Feature-based
 Intensity

CS4243

gradient-based

Motion Tracking

15

Feature-based Motion Tracking


Look for distinct features that change positions.

 Eagle’s

wing tips change positions.
 Tree tops don’t change positions.
CS4243

Motion Tracking

16

Basic Ideas
1.

Look for distinct features in current frame.

2.

For each feature


Search for matching feature within neighbourhood
in next frame.
 Difference in positions  displacement.
 Velocity = displacement / time difference.

CS4243

Motion Tracking

17

Basic Ideas
displacement

feature area

search area

CS4243

Motion Tracking

18

What feature to use?


Harris corner



Tomasi’s feature



Feature descriptors: SIFT, SURF, GLOH, etc.



Others

CS4243

Motion Tracking

19

Summary


Simple algorithm.



Can be slow if search area is large.



Can constrain search area with prior knowledge.

CS4243

Motion Tracking

20

Gradient-based Motion Tracking


Two basic assumptions
 Intensity

changes smoothly with position.
 Intensity of object doesn’t change over time.


Suppose an object is in motion.
 Change

position (dx, dy) over time dt.
 Then, from 2nd assumption:
 Apply

CS4243

Taylor’s series expansion:

Motion Tracking

21

 Omit

higher order terms and divide by dt

 Denote

 Then,

 u,

v are unknown.
 2 unknowns, 1 equation: can’t solve!

CS4243

Motion Tracking

22

Any difference in motion?

CS4243

Motion Tracking

23

Any difference in motion?

CS4243

Motion Tracking

24

Any difference in motion?

CS4243

Motion Tracking

25

Any difference in motion?

CS4243

Motion Tracking

26

Any difference in motion?

CS4243

Motion Tracking

27

Any difference in motion?

CS4243

Motion Tracking

28

Aperture Problem


Homogeneous region

 Ix

= Iy = It = 0.
 No change in local region.
 Cannot detect motion.

CS4243

Motion Tracking

29

Aperture Problem


Edge

 Ix

and Iy are zero along edge
 Cannot measure motion tangential to edge
 Ix and Iy are non-zero normal to edge
 Can measure motion normal to edge
 So, cannot measure actual motion

CS4243

Motion Tracking

30

Aperture Problem


Corner

 Ix

and Iy are non-zero in two perpendicular directions
 2 unknowns, 2 equations
 Can measure actual motion

CS4243

Motion Tracking

31

Lucas-Kanade Method


Consider two consecutive image frames I and J:
I

J
x+d
d
x

x

moves from x = (x, y)T to x + d.
 d = (u, v) T
 Object

CS4243

Motion Tracking

32



So,



Or



Due to noise, there’s an error at position x:



Sum error over small window W at position x:

Similar to
template matching
CS4243

weight
Motion Tracking

33



If E is small, patterns in I and J match well.



So, find d that minimises E:
E / d = 0, compute d that minimises E.
 First, expand I(x – d) by Taylor’s series expansion:
 Set

 Omit

higher order terms:

 Write

in matrix form:

Intensity gradient
CS4243

Motion Tracking

34

 Now,

error E at position x is:

 Now,

differentiate E with respect to d (exercise):

 Setting

E / d = 0 gives

b

the only
unknown

Z

CS4243

Motion Tracking

35

 So,

2



CS4243

we get

unknowns, 2 equations. Can solve for d = (u, v).

What happen to aperture problem?
Did it disappear?
Motion Tracking

36

Lucas-Kanade + Tomasi


Lucas-Kanade algorithm is often used with
Tomasi’s feature
 Apply

Tomasi’s method to detect good features.
 Apply LK method to compute d for each pixel.
 Accept d only for good features.

CS4243

Motion Tracking

37

Example

Can you spot tracking errors?

CS4243

Motion Tracking

38

Constraints


Math of LK tracker assumes d is small.



In implementation, W is also small.



LK tracker is good only for small displacement.

CS4243

Motion Tracking

39

How to handle large displacement?


What if we scale down images?
 Displacements

CS4243

are smaller!

Motion Tracking

40

Image Pyramid
Usually, smoothen
image with Gaussian
filter before scaling
down.

downsample

scaled image

input image
CS4243

Motion Tracking

41

LK Tracking with Image Pyramid

CS4243

Motion Tracking

42



Construct image pyramids.



Apply LK tracker to low-resolution images.



Propagate results to higher-resolution images.



Apply LK tracker to higher-resolution images.

CS4243

Motion Tracking

43

Example

Tracking results are more accurate.

CS4243

Motion Tracking

44

Summary


Efficient algorithm, no explicit search.



Has aperture problem;
track good features only



LK tracker can’t track large displacement.



Use LK + image pyramid for large displacement.

CS4243

Motion Tracking

45

Software


OpenCV supports LK and LK with pyramid.



[Bir] offers LK with Tomasi’s features & pyramid.

CS4243

Motion Tracking

46

Appendix


Calculation of Ix, Iy, It
 Use

finite difference method
 Forward difference
Ix = I(x  1, y, t)  I(x, y, t)
Iy = I(x, y  1, t)  I(x, y, t)
It = I(x, y, t  1)  I(x, y, t)
 Backward difference
Ix = I(x, y, t)  I(x  1, y, t)
Iy = I(x, y, t)  I(x, y  1, t)
It = I(x, y, t)  I(x, y, t  1)

CS4243

Motion Tracking

47

Further Readings


Lucas-Kanade tracking with pyramid:
[BK08] Chapter 10.



Optical flow: [Sze10] Section 8.4.



Hierarchical motion estimation (with image
pyramid): [Sze10] Section 8.1.1.

CS4243

Motion Tracking

48

References


[Bir] S. Birchfield. KLT: An implementation of the Kanade-LucasTomasi feature tracker. http://vision.stanford.edu/~birch/klt/.



[BK08] Bradski and Kaehler. Learning OpenCV: Computer Vision
with the OpenCV Library. O’Reilly, 2008.



[LK81] B. D. Lucas and T. Kanade. An iterative image registration
technique with an application to stereo vision. In Proceedings of 7th
International Joint Conference on Artificial Intelligence, pages 674–
679, 1981.



[ST94] J. Shi and C. Tomasi. Good features to track. In Proceedings
of IEEE Conference on Computer Vision and Pattern Recognition,
pages 593–600, 1994.



[Sze10] R. Szeliski. Computer Vision: Algorithms and Applications.
Springer, 2010.

CS4243

Motion Tracking

49

References


CS4243

[TK91] C. Tomasi and T. Kanade. Detection and tracking of point
features. Technical Report CMU-CS-91-132, School of Computer
Science, Carnegie Mellon University, 1991.

Motion Tracking

50


Slide 40

Leow Wee Kheng
CS4243 Computer Vision and Pattern Recognition

Motion Tracking

CS4243

Motion Tracking

1

Changes are everywhere!

CS4243

Motion Tracking

2

Illumination change

CS4243

Motion Tracking

3

Shape change

CS4243

Motion Tracking

4

Object motion

CS4243

Motion Tracking

5

Camera motion

CS4243

Motion Tracking

6

Object & camera motion

CS4243

Motion Tracking

7

Everything changes!

CS4243

Motion Tracking

8

Motion analysis is tough in general!

We focus on object / camera motion.

CS4243

Motion Tracking

9

Change Detection


Detects any change in two video frames.



Straightforward method:
 Compute

difference between corresponding pixels:



: intensity / colour at (x, y) in frame t.
> threshold, has large difference.

 If

CS4243

Motion Tracking

10

Any difference?

No

CS4243

Motion Tracking

11

Any difference?

Yes,
illumination
change

CS4243

Motion Tracking

12

Any difference?

Yes,
position change

CS4243

Motion Tracking

13

Change Detection


Can detect
 Illumination

change
 Position change
 Illumination and position change


But, cannot distinguish between them.



Need to detect and measure position change.

CS4243

Motion Tracking

14

Motion Tracking


Two approaches
 Feature-based
 Intensity

CS4243

gradient-based

Motion Tracking

15

Feature-based Motion Tracking


Look for distinct features that change positions.

 Eagle’s

wing tips change positions.
 Tree tops don’t change positions.
CS4243

Motion Tracking

16

Basic Ideas
1.

Look for distinct features in current frame.

2.

For each feature


Search for matching feature within neighbourhood
in next frame.
 Difference in positions  displacement.
 Velocity = displacement / time difference.

CS4243

Motion Tracking

17

Basic Ideas
displacement

feature area

search area

CS4243

Motion Tracking

18

What feature to use?


Harris corner



Tomasi’s feature



Feature descriptors: SIFT, SURF, GLOH, etc.



Others

CS4243

Motion Tracking

19

Summary


Simple algorithm.



Can be slow if search area is large.



Can constrain search area with prior knowledge.

CS4243

Motion Tracking

20

Gradient-based Motion Tracking


Two basic assumptions
 Intensity

changes smoothly with position.
 Intensity of object doesn’t change over time.


Suppose an object is in motion.
 Change

position (dx, dy) over time dt.
 Then, from 2nd assumption:
 Apply

CS4243

Taylor’s series expansion:

Motion Tracking

21

 Omit

higher order terms and divide by dt

 Denote

 Then,

 u,

v are unknown.
 2 unknowns, 1 equation: can’t solve!

CS4243

Motion Tracking

22

Any difference in motion?

CS4243

Motion Tracking

23

Any difference in motion?

CS4243

Motion Tracking

24

Any difference in motion?

CS4243

Motion Tracking

25

Any difference in motion?

CS4243

Motion Tracking

26

Any difference in motion?

CS4243

Motion Tracking

27

Any difference in motion?

CS4243

Motion Tracking

28

Aperture Problem


Homogeneous region

 Ix

= Iy = It = 0.
 No change in local region.
 Cannot detect motion.

CS4243

Motion Tracking

29

Aperture Problem


Edge

 Ix

and Iy are zero along edge
 Cannot measure motion tangential to edge
 Ix and Iy are non-zero normal to edge
 Can measure motion normal to edge
 So, cannot measure actual motion

CS4243

Motion Tracking

30

Aperture Problem


Corner

 Ix

and Iy are non-zero in two perpendicular directions
 2 unknowns, 2 equations
 Can measure actual motion

CS4243

Motion Tracking

31

Lucas-Kanade Method


Consider two consecutive image frames I and J:
I

J
x+d
d
x

x

moves from x = (x, y)T to x + d.
 d = (u, v) T
 Object

CS4243

Motion Tracking

32



So,



Or



Due to noise, there’s an error at position x:



Sum error over small window W at position x:

Similar to
template matching
CS4243

weight
Motion Tracking

33



If E is small, patterns in I and J match well.



So, find d that minimises E:
E / d = 0, compute d that minimises E.
 First, expand I(x – d) by Taylor’s series expansion:
 Set

 Omit

higher order terms:

 Write

in matrix form:

Intensity gradient
CS4243

Motion Tracking

34

 Now,

error E at position x is:

 Now,

differentiate E with respect to d (exercise):

 Setting

E / d = 0 gives

b

the only
unknown

Z

CS4243

Motion Tracking

35

 So,

2



CS4243

we get

unknowns, 2 equations. Can solve for d = (u, v).

What happen to aperture problem?
Did it disappear?
Motion Tracking

36

Lucas-Kanade + Tomasi


Lucas-Kanade algorithm is often used with
Tomasi’s feature
 Apply

Tomasi’s method to detect good features.
 Apply LK method to compute d for each pixel.
 Accept d only for good features.

CS4243

Motion Tracking

37

Example

Can you spot tracking errors?

CS4243

Motion Tracking

38

Constraints


Math of LK tracker assumes d is small.



In implementation, W is also small.



LK tracker is good only for small displacement.

CS4243

Motion Tracking

39

How to handle large displacement?


What if we scale down images?
 Displacements

CS4243

are smaller!

Motion Tracking

40

Image Pyramid
Usually, smoothen
image with Gaussian
filter before scaling
down.

downsample

scaled image

input image
CS4243

Motion Tracking

41

LK Tracking with Image Pyramid

CS4243

Motion Tracking

42



Construct image pyramids.



Apply LK tracker to low-resolution images.



Propagate results to higher-resolution images.



Apply LK tracker to higher-resolution images.

CS4243

Motion Tracking

43

Example

Tracking results are more accurate.

CS4243

Motion Tracking

44

Summary


Efficient algorithm, no explicit search.



Has aperture problem;
track good features only



LK tracker can’t track large displacement.



Use LK + image pyramid for large displacement.

CS4243

Motion Tracking

45

Software


OpenCV supports LK and LK with pyramid.



[Bir] offers LK with Tomasi’s features & pyramid.

CS4243

Motion Tracking

46

Appendix


Calculation of Ix, Iy, It
 Use

finite difference method
 Forward difference
Ix = I(x  1, y, t)  I(x, y, t)
Iy = I(x, y  1, t)  I(x, y, t)
It = I(x, y, t  1)  I(x, y, t)
 Backward difference
Ix = I(x, y, t)  I(x  1, y, t)
Iy = I(x, y, t)  I(x, y  1, t)
It = I(x, y, t)  I(x, y, t  1)

CS4243

Motion Tracking

47

Further Readings


Lucas-Kanade tracking with pyramid:
[BK08] Chapter 10.



Optical flow: [Sze10] Section 8.4.



Hierarchical motion estimation (with image
pyramid): [Sze10] Section 8.1.1.

CS4243

Motion Tracking

48

References


[Bir] S. Birchfield. KLT: An implementation of the Kanade-LucasTomasi feature tracker. http://vision.stanford.edu/~birch/klt/.



[BK08] Bradski and Kaehler. Learning OpenCV: Computer Vision
with the OpenCV Library. O’Reilly, 2008.



[LK81] B. D. Lucas and T. Kanade. An iterative image registration
technique with an application to stereo vision. In Proceedings of 7th
International Joint Conference on Artificial Intelligence, pages 674–
679, 1981.



[ST94] J. Shi and C. Tomasi. Good features to track. In Proceedings
of IEEE Conference on Computer Vision and Pattern Recognition,
pages 593–600, 1994.



[Sze10] R. Szeliski. Computer Vision: Algorithms and Applications.
Springer, 2010.

CS4243

Motion Tracking

49

References


CS4243

[TK91] C. Tomasi and T. Kanade. Detection and tracking of point
features. Technical Report CMU-CS-91-132, School of Computer
Science, Carnegie Mellon University, 1991.

Motion Tracking

50


Slide 41

Leow Wee Kheng
CS4243 Computer Vision and Pattern Recognition

Motion Tracking

CS4243

Motion Tracking

1

Changes are everywhere!

CS4243

Motion Tracking

2

Illumination change

CS4243

Motion Tracking

3

Shape change

CS4243

Motion Tracking

4

Object motion

CS4243

Motion Tracking

5

Camera motion

CS4243

Motion Tracking

6

Object & camera motion

CS4243

Motion Tracking

7

Everything changes!

CS4243

Motion Tracking

8

Motion analysis is tough in general!

We focus on object / camera motion.

CS4243

Motion Tracking

9

Change Detection


Detects any change in two video frames.



Straightforward method:
 Compute

difference between corresponding pixels:



: intensity / colour at (x, y) in frame t.
> threshold, has large difference.

 If

CS4243

Motion Tracking

10

Any difference?

No

CS4243

Motion Tracking

11

Any difference?

Yes,
illumination
change

CS4243

Motion Tracking

12

Any difference?

Yes,
position change

CS4243

Motion Tracking

13

Change Detection


Can detect
 Illumination

change
 Position change
 Illumination and position change


But, cannot distinguish between them.



Need to detect and measure position change.

CS4243

Motion Tracking

14

Motion Tracking


Two approaches
 Feature-based
 Intensity

CS4243

gradient-based

Motion Tracking

15

Feature-based Motion Tracking


Look for distinct features that change positions.

 Eagle’s

wing tips change positions.
 Tree tops don’t change positions.
CS4243

Motion Tracking

16

Basic Ideas
1.

Look for distinct features in current frame.

2.

For each feature


Search for matching feature within neighbourhood
in next frame.
 Difference in positions  displacement.
 Velocity = displacement / time difference.

CS4243

Motion Tracking

17

Basic Ideas
displacement

feature area

search area

CS4243

Motion Tracking

18

What feature to use?


Harris corner



Tomasi’s feature



Feature descriptors: SIFT, SURF, GLOH, etc.



Others

CS4243

Motion Tracking

19

Summary


Simple algorithm.



Can be slow if search area is large.



Can constrain search area with prior knowledge.

CS4243

Motion Tracking

20

Gradient-based Motion Tracking


Two basic assumptions
 Intensity

changes smoothly with position.
 Intensity of object doesn’t change over time.


Suppose an object is in motion.
 Change

position (dx, dy) over time dt.
 Then, from 2nd assumption:
 Apply

CS4243

Taylor’s series expansion:

Motion Tracking

21

 Omit

higher order terms and divide by dt

 Denote

 Then,

 u,

v are unknown.
 2 unknowns, 1 equation: can’t solve!

CS4243

Motion Tracking

22

Any difference in motion?

CS4243

Motion Tracking

23

Any difference in motion?

CS4243

Motion Tracking

24

Any difference in motion?

CS4243

Motion Tracking

25

Any difference in motion?

CS4243

Motion Tracking

26

Any difference in motion?

CS4243

Motion Tracking

27

Any difference in motion?

CS4243

Motion Tracking

28

Aperture Problem


Homogeneous region

 Ix

= Iy = It = 0.
 No change in local region.
 Cannot detect motion.

CS4243

Motion Tracking

29

Aperture Problem


Edge

 Ix

and Iy are zero along edge
 Cannot measure motion tangential to edge
 Ix and Iy are non-zero normal to edge
 Can measure motion normal to edge
 So, cannot measure actual motion

CS4243

Motion Tracking

30

Aperture Problem


Corner

 Ix

and Iy are non-zero in two perpendicular directions
 2 unknowns, 2 equations
 Can measure actual motion

CS4243

Motion Tracking

31

Lucas-Kanade Method


Consider two consecutive image frames I and J:
I

J
x+d
d
x

x

moves from x = (x, y)T to x + d.
 d = (u, v) T
 Object

CS4243

Motion Tracking

32



So,



Or



Due to noise, there’s an error at position x:



Sum error over small window W at position x:

Similar to
template matching
CS4243

weight
Motion Tracking

33



If E is small, patterns in I and J match well.



So, find d that minimises E:
E / d = 0, compute d that minimises E.
 First, expand I(x – d) by Taylor’s series expansion:
 Set

 Omit

higher order terms:

 Write

in matrix form:

Intensity gradient
CS4243

Motion Tracking

34

 Now,

error E at position x is:

 Now,

differentiate E with respect to d (exercise):

 Setting

E / d = 0 gives

b

the only
unknown

Z

CS4243

Motion Tracking

35

 So,

2



CS4243

we get

unknowns, 2 equations. Can solve for d = (u, v).

What happen to aperture problem?
Did it disappear?
Motion Tracking

36

Lucas-Kanade + Tomasi


Lucas-Kanade algorithm is often used with
Tomasi’s feature
 Apply

Tomasi’s method to detect good features.
 Apply LK method to compute d for each pixel.
 Accept d only for good features.

CS4243

Motion Tracking

37

Example

Can you spot tracking errors?

CS4243

Motion Tracking

38

Constraints


Math of LK tracker assumes d is small.



In implementation, W is also small.



LK tracker is good only for small displacement.

CS4243

Motion Tracking

39

How to handle large displacement?


What if we scale down images?
 Displacements

CS4243

are smaller!

Motion Tracking

40

Image Pyramid
Usually, smoothen
image with Gaussian
filter before scaling
down.

downsample

scaled image

input image
CS4243

Motion Tracking

41

LK Tracking with Image Pyramid

CS4243

Motion Tracking

42



Construct image pyramids.



Apply LK tracker to low-resolution images.



Propagate results to higher-resolution images.



Apply LK tracker to higher-resolution images.

CS4243

Motion Tracking

43

Example

Tracking results are more accurate.

CS4243

Motion Tracking

44

Summary


Efficient algorithm, no explicit search.



Has aperture problem;
track good features only



LK tracker can’t track large displacement.



Use LK + image pyramid for large displacement.

CS4243

Motion Tracking

45

Software


OpenCV supports LK and LK with pyramid.



[Bir] offers LK with Tomasi’s features & pyramid.

CS4243

Motion Tracking

46

Appendix


Calculation of Ix, Iy, It
 Use

finite difference method
 Forward difference
Ix = I(x  1, y, t)  I(x, y, t)
Iy = I(x, y  1, t)  I(x, y, t)
It = I(x, y, t  1)  I(x, y, t)
 Backward difference
Ix = I(x, y, t)  I(x  1, y, t)
Iy = I(x, y, t)  I(x, y  1, t)
It = I(x, y, t)  I(x, y, t  1)

CS4243

Motion Tracking

47

Further Readings


Lucas-Kanade tracking with pyramid:
[BK08] Chapter 10.



Optical flow: [Sze10] Section 8.4.



Hierarchical motion estimation (with image
pyramid): [Sze10] Section 8.1.1.

CS4243

Motion Tracking

48

References


[Bir] S. Birchfield. KLT: An implementation of the Kanade-LucasTomasi feature tracker. http://vision.stanford.edu/~birch/klt/.



[BK08] Bradski and Kaehler. Learning OpenCV: Computer Vision
with the OpenCV Library. O’Reilly, 2008.



[LK81] B. D. Lucas and T. Kanade. An iterative image registration
technique with an application to stereo vision. In Proceedings of 7th
International Joint Conference on Artificial Intelligence, pages 674–
679, 1981.



[ST94] J. Shi and C. Tomasi. Good features to track. In Proceedings
of IEEE Conference on Computer Vision and Pattern Recognition,
pages 593–600, 1994.



[Sze10] R. Szeliski. Computer Vision: Algorithms and Applications.
Springer, 2010.

CS4243

Motion Tracking

49

References


CS4243

[TK91] C. Tomasi and T. Kanade. Detection and tracking of point
features. Technical Report CMU-CS-91-132, School of Computer
Science, Carnegie Mellon University, 1991.

Motion Tracking

50


Slide 42

Leow Wee Kheng
CS4243 Computer Vision and Pattern Recognition

Motion Tracking

CS4243

Motion Tracking

1

Changes are everywhere!

CS4243

Motion Tracking

2

Illumination change

CS4243

Motion Tracking

3

Shape change

CS4243

Motion Tracking

4

Object motion

CS4243

Motion Tracking

5

Camera motion

CS4243

Motion Tracking

6

Object & camera motion

CS4243

Motion Tracking

7

Everything changes!

CS4243

Motion Tracking

8

Motion analysis is tough in general!

We focus on object / camera motion.

CS4243

Motion Tracking

9

Change Detection


Detects any change in two video frames.



Straightforward method:
 Compute

difference between corresponding pixels:



: intensity / colour at (x, y) in frame t.
> threshold, has large difference.

 If

CS4243

Motion Tracking

10

Any difference?

No

CS4243

Motion Tracking

11

Any difference?

Yes,
illumination
change

CS4243

Motion Tracking

12

Any difference?

Yes,
position change

CS4243

Motion Tracking

13

Change Detection


Can detect
 Illumination

change
 Position change
 Illumination and position change


But, cannot distinguish between them.



Need to detect and measure position change.

CS4243

Motion Tracking

14

Motion Tracking


Two approaches
 Feature-based
 Intensity

CS4243

gradient-based

Motion Tracking

15

Feature-based Motion Tracking


Look for distinct features that change positions.

 Eagle’s

wing tips change positions.
 Tree tops don’t change positions.
CS4243

Motion Tracking

16

Basic Ideas
1.

Look for distinct features in current frame.

2.

For each feature


Search for matching feature within neighbourhood
in next frame.
 Difference in positions  displacement.
 Velocity = displacement / time difference.

CS4243

Motion Tracking

17

Basic Ideas
displacement

feature area

search area

CS4243

Motion Tracking

18

What feature to use?


Harris corner



Tomasi’s feature



Feature descriptors: SIFT, SURF, GLOH, etc.



Others

CS4243

Motion Tracking

19

Summary


Simple algorithm.



Can be slow if search area is large.



Can constrain search area with prior knowledge.

CS4243

Motion Tracking

20

Gradient-based Motion Tracking


Two basic assumptions
 Intensity

changes smoothly with position.
 Intensity of object doesn’t change over time.


Suppose an object is in motion.
 Change

position (dx, dy) over time dt.
 Then, from 2nd assumption:
 Apply

CS4243

Taylor’s series expansion:

Motion Tracking

21

 Omit

higher order terms and divide by dt

 Denote

 Then,

 u,

v are unknown.
 2 unknowns, 1 equation: can’t solve!

CS4243

Motion Tracking

22

Any difference in motion?

CS4243

Motion Tracking

23

Any difference in motion?

CS4243

Motion Tracking

24

Any difference in motion?

CS4243

Motion Tracking

25

Any difference in motion?

CS4243

Motion Tracking

26

Any difference in motion?

CS4243

Motion Tracking

27

Any difference in motion?

CS4243

Motion Tracking

28

Aperture Problem


Homogeneous region

 Ix

= Iy = It = 0.
 No change in local region.
 Cannot detect motion.

CS4243

Motion Tracking

29

Aperture Problem


Edge

 Ix

and Iy are zero along edge
 Cannot measure motion tangential to edge
 Ix and Iy are non-zero normal to edge
 Can measure motion normal to edge
 So, cannot measure actual motion

CS4243

Motion Tracking

30

Aperture Problem


Corner

 Ix

and Iy are non-zero in two perpendicular directions
 2 unknowns, 2 equations
 Can measure actual motion

CS4243

Motion Tracking

31

Lucas-Kanade Method


Consider two consecutive image frames I and J:
I

J
x+d
d
x

x

moves from x = (x, y)T to x + d.
 d = (u, v) T
 Object

CS4243

Motion Tracking

32



So,



Or



Due to noise, there’s an error at position x:



Sum error over small window W at position x:

Similar to
template matching
CS4243

weight
Motion Tracking

33



If E is small, patterns in I and J match well.



So, find d that minimises E:
E / d = 0, compute d that minimises E.
 First, expand I(x – d) by Taylor’s series expansion:
 Set

 Omit

higher order terms:

 Write

in matrix form:

Intensity gradient
CS4243

Motion Tracking

34

 Now,

error E at position x is:

 Now,

differentiate E with respect to d (exercise):

 Setting

E / d = 0 gives

b

the only
unknown

Z

CS4243

Motion Tracking

35

 So,

2



CS4243

we get

unknowns, 2 equations. Can solve for d = (u, v).

What happen to aperture problem?
Did it disappear?
Motion Tracking

36

Lucas-Kanade + Tomasi


Lucas-Kanade algorithm is often used with
Tomasi’s feature
 Apply

Tomasi’s method to detect good features.
 Apply LK method to compute d for each pixel.
 Accept d only for good features.

CS4243

Motion Tracking

37

Example

Can you spot tracking errors?

CS4243

Motion Tracking

38

Constraints


Math of LK tracker assumes d is small.



In implementation, W is also small.



LK tracker is good only for small displacement.

CS4243

Motion Tracking

39

How to handle large displacement?


What if we scale down images?
 Displacements

CS4243

are smaller!

Motion Tracking

40

Image Pyramid
Usually, smoothen
image with Gaussian
filter before scaling
down.

downsample

scaled image

input image
CS4243

Motion Tracking

41

LK Tracking with Image Pyramid

CS4243

Motion Tracking

42



Construct image pyramids.



Apply LK tracker to low-resolution images.



Propagate results to higher-resolution images.



Apply LK tracker to higher-resolution images.

CS4243

Motion Tracking

43

Example

Tracking results are more accurate.

CS4243

Motion Tracking

44

Summary


Efficient algorithm, no explicit search.



Has aperture problem;
track good features only



LK tracker can’t track large displacement.



Use LK + image pyramid for large displacement.

CS4243

Motion Tracking

45

Software


OpenCV supports LK and LK with pyramid.



[Bir] offers LK with Tomasi’s features & pyramid.

CS4243

Motion Tracking

46

Appendix


Calculation of Ix, Iy, It
 Use

finite difference method
 Forward difference
Ix = I(x  1, y, t)  I(x, y, t)
Iy = I(x, y  1, t)  I(x, y, t)
It = I(x, y, t  1)  I(x, y, t)
 Backward difference
Ix = I(x, y, t)  I(x  1, y, t)
Iy = I(x, y, t)  I(x, y  1, t)
It = I(x, y, t)  I(x, y, t  1)

CS4243

Motion Tracking

47

Further Readings


Lucas-Kanade tracking with pyramid:
[BK08] Chapter 10.



Optical flow: [Sze10] Section 8.4.



Hierarchical motion estimation (with image
pyramid): [Sze10] Section 8.1.1.

CS4243

Motion Tracking

48

References


[Bir] S. Birchfield. KLT: An implementation of the Kanade-LucasTomasi feature tracker. http://vision.stanford.edu/~birch/klt/.



[BK08] Bradski and Kaehler. Learning OpenCV: Computer Vision
with the OpenCV Library. O’Reilly, 2008.



[LK81] B. D. Lucas and T. Kanade. An iterative image registration
technique with an application to stereo vision. In Proceedings of 7th
International Joint Conference on Artificial Intelligence, pages 674–
679, 1981.



[ST94] J. Shi and C. Tomasi. Good features to track. In Proceedings
of IEEE Conference on Computer Vision and Pattern Recognition,
pages 593–600, 1994.



[Sze10] R. Szeliski. Computer Vision: Algorithms and Applications.
Springer, 2010.

CS4243

Motion Tracking

49

References


CS4243

[TK91] C. Tomasi and T. Kanade. Detection and tracking of point
features. Technical Report CMU-CS-91-132, School of Computer
Science, Carnegie Mellon University, 1991.

Motion Tracking

50


Slide 43

Leow Wee Kheng
CS4243 Computer Vision and Pattern Recognition

Motion Tracking

CS4243

Motion Tracking

1

Changes are everywhere!

CS4243

Motion Tracking

2

Illumination change

CS4243

Motion Tracking

3

Shape change

CS4243

Motion Tracking

4

Object motion

CS4243

Motion Tracking

5

Camera motion

CS4243

Motion Tracking

6

Object & camera motion

CS4243

Motion Tracking

7

Everything changes!

CS4243

Motion Tracking

8

Motion analysis is tough in general!

We focus on object / camera motion.

CS4243

Motion Tracking

9

Change Detection


Detects any change in two video frames.



Straightforward method:
 Compute

difference between corresponding pixels:



: intensity / colour at (x, y) in frame t.
> threshold, has large difference.

 If

CS4243

Motion Tracking

10

Any difference?

No

CS4243

Motion Tracking

11

Any difference?

Yes,
illumination
change

CS4243

Motion Tracking

12

Any difference?

Yes,
position change

CS4243

Motion Tracking

13

Change Detection


Can detect
 Illumination

change
 Position change
 Illumination and position change


But, cannot distinguish between them.



Need to detect and measure position change.

CS4243

Motion Tracking

14

Motion Tracking


Two approaches
 Feature-based
 Intensity

CS4243

gradient-based

Motion Tracking

15

Feature-based Motion Tracking


Look for distinct features that change positions.

 Eagle’s

wing tips change positions.
 Tree tops don’t change positions.
CS4243

Motion Tracking

16

Basic Ideas
1.

Look for distinct features in current frame.

2.

For each feature


Search for matching feature within neighbourhood
in next frame.
 Difference in positions  displacement.
 Velocity = displacement / time difference.

CS4243

Motion Tracking

17

Basic Ideas
displacement

feature area

search area

CS4243

Motion Tracking

18

What feature to use?


Harris corner



Tomasi’s feature



Feature descriptors: SIFT, SURF, GLOH, etc.



Others

CS4243

Motion Tracking

19

Summary


Simple algorithm.



Can be slow if search area is large.



Can constrain search area with prior knowledge.

CS4243

Motion Tracking

20

Gradient-based Motion Tracking


Two basic assumptions
 Intensity

changes smoothly with position.
 Intensity of object doesn’t change over time.


Suppose an object is in motion.
 Change

position (dx, dy) over time dt.
 Then, from 2nd assumption:
 Apply

CS4243

Taylor’s series expansion:

Motion Tracking

21

 Omit

higher order terms and divide by dt

 Denote

 Then,

 u,

v are unknown.
 2 unknowns, 1 equation: can’t solve!

CS4243

Motion Tracking

22

Any difference in motion?

CS4243

Motion Tracking

23

Any difference in motion?

CS4243

Motion Tracking

24

Any difference in motion?

CS4243

Motion Tracking

25

Any difference in motion?

CS4243

Motion Tracking

26

Any difference in motion?

CS4243

Motion Tracking

27

Any difference in motion?

CS4243

Motion Tracking

28

Aperture Problem


Homogeneous region

 Ix

= Iy = It = 0.
 No change in local region.
 Cannot detect motion.

CS4243

Motion Tracking

29

Aperture Problem


Edge

 Ix

and Iy are zero along edge
 Cannot measure motion tangential to edge
 Ix and Iy are non-zero normal to edge
 Can measure motion normal to edge
 So, cannot measure actual motion

CS4243

Motion Tracking

30

Aperture Problem


Corner

 Ix

and Iy are non-zero in two perpendicular directions
 2 unknowns, 2 equations
 Can measure actual motion

CS4243

Motion Tracking

31

Lucas-Kanade Method


Consider two consecutive image frames I and J:
I

J
x+d
d
x

x

moves from x = (x, y)T to x + d.
 d = (u, v) T
 Object

CS4243

Motion Tracking

32



So,



Or



Due to noise, there’s an error at position x:



Sum error over small window W at position x:

Similar to
template matching
CS4243

weight
Motion Tracking

33



If E is small, patterns in I and J match well.



So, find d that minimises E:
E / d = 0, compute d that minimises E.
 First, expand I(x – d) by Taylor’s series expansion:
 Set

 Omit

higher order terms:

 Write

in matrix form:

Intensity gradient
CS4243

Motion Tracking

34

 Now,

error E at position x is:

 Now,

differentiate E with respect to d (exercise):

 Setting

E / d = 0 gives

b

the only
unknown

Z

CS4243

Motion Tracking

35

 So,

2



CS4243

we get

unknowns, 2 equations. Can solve for d = (u, v).

What happen to aperture problem?
Did it disappear?
Motion Tracking

36

Lucas-Kanade + Tomasi


Lucas-Kanade algorithm is often used with
Tomasi’s feature
 Apply

Tomasi’s method to detect good features.
 Apply LK method to compute d for each pixel.
 Accept d only for good features.

CS4243

Motion Tracking

37

Example

Can you spot tracking errors?

CS4243

Motion Tracking

38

Constraints


Math of LK tracker assumes d is small.



In implementation, W is also small.



LK tracker is good only for small displacement.

CS4243

Motion Tracking

39

How to handle large displacement?


What if we scale down images?
 Displacements

CS4243

are smaller!

Motion Tracking

40

Image Pyramid
Usually, smoothen
image with Gaussian
filter before scaling
down.

downsample

scaled image

input image
CS4243

Motion Tracking

41

LK Tracking with Image Pyramid

CS4243

Motion Tracking

42



Construct image pyramids.



Apply LK tracker to low-resolution images.



Propagate results to higher-resolution images.



Apply LK tracker to higher-resolution images.

CS4243

Motion Tracking

43

Example

Tracking results are more accurate.

CS4243

Motion Tracking

44

Summary


Efficient algorithm, no explicit search.



Has aperture problem;
track good features only



LK tracker can’t track large displacement.



Use LK + image pyramid for large displacement.

CS4243

Motion Tracking

45

Software


OpenCV supports LK and LK with pyramid.



[Bir] offers LK with Tomasi’s features & pyramid.

CS4243

Motion Tracking

46

Appendix


Calculation of Ix, Iy, It
 Use

finite difference method
 Forward difference
Ix = I(x  1, y, t)  I(x, y, t)
Iy = I(x, y  1, t)  I(x, y, t)
It = I(x, y, t  1)  I(x, y, t)
 Backward difference
Ix = I(x, y, t)  I(x  1, y, t)
Iy = I(x, y, t)  I(x, y  1, t)
It = I(x, y, t)  I(x, y, t  1)

CS4243

Motion Tracking

47

Further Readings


Lucas-Kanade tracking with pyramid:
[BK08] Chapter 10.



Optical flow: [Sze10] Section 8.4.



Hierarchical motion estimation (with image
pyramid): [Sze10] Section 8.1.1.

CS4243

Motion Tracking

48

References


[Bir] S. Birchfield. KLT: An implementation of the Kanade-LucasTomasi feature tracker. http://vision.stanford.edu/~birch/klt/.



[BK08] Bradski and Kaehler. Learning OpenCV: Computer Vision
with the OpenCV Library. O’Reilly, 2008.



[LK81] B. D. Lucas and T. Kanade. An iterative image registration
technique with an application to stereo vision. In Proceedings of 7th
International Joint Conference on Artificial Intelligence, pages 674–
679, 1981.



[ST94] J. Shi and C. Tomasi. Good features to track. In Proceedings
of IEEE Conference on Computer Vision and Pattern Recognition,
pages 593–600, 1994.



[Sze10] R. Szeliski. Computer Vision: Algorithms and Applications.
Springer, 2010.

CS4243

Motion Tracking

49

References


CS4243

[TK91] C. Tomasi and T. Kanade. Detection and tracking of point
features. Technical Report CMU-CS-91-132, School of Computer
Science, Carnegie Mellon University, 1991.

Motion Tracking

50


Slide 44

Leow Wee Kheng
CS4243 Computer Vision and Pattern Recognition

Motion Tracking

CS4243

Motion Tracking

1

Changes are everywhere!

CS4243

Motion Tracking

2

Illumination change

CS4243

Motion Tracking

3

Shape change

CS4243

Motion Tracking

4

Object motion

CS4243

Motion Tracking

5

Camera motion

CS4243

Motion Tracking

6

Object & camera motion

CS4243

Motion Tracking

7

Everything changes!

CS4243

Motion Tracking

8

Motion analysis is tough in general!

We focus on object / camera motion.

CS4243

Motion Tracking

9

Change Detection


Detects any change in two video frames.



Straightforward method:
 Compute

difference between corresponding pixels:



: intensity / colour at (x, y) in frame t.
> threshold, has large difference.

 If

CS4243

Motion Tracking

10

Any difference?

No

CS4243

Motion Tracking

11

Any difference?

Yes,
illumination
change

CS4243

Motion Tracking

12

Any difference?

Yes,
position change

CS4243

Motion Tracking

13

Change Detection


Can detect
 Illumination

change
 Position change
 Illumination and position change


But, cannot distinguish between them.



Need to detect and measure position change.

CS4243

Motion Tracking

14

Motion Tracking


Two approaches
 Feature-based
 Intensity

CS4243

gradient-based

Motion Tracking

15

Feature-based Motion Tracking


Look for distinct features that change positions.

 Eagle’s

wing tips change positions.
 Tree tops don’t change positions.
CS4243

Motion Tracking

16

Basic Ideas
1.

Look for distinct features in current frame.

2.

For each feature


Search for matching feature within neighbourhood
in next frame.
 Difference in positions  displacement.
 Velocity = displacement / time difference.

CS4243

Motion Tracking

17

Basic Ideas
displacement

feature area

search area

CS4243

Motion Tracking

18

What feature to use?


Harris corner



Tomasi’s feature



Feature descriptors: SIFT, SURF, GLOH, etc.



Others

CS4243

Motion Tracking

19

Summary


Simple algorithm.



Can be slow if search area is large.



Can constrain search area with prior knowledge.

CS4243

Motion Tracking

20

Gradient-based Motion Tracking


Two basic assumptions
 Intensity

changes smoothly with position.
 Intensity of object doesn’t change over time.


Suppose an object is in motion.
 Change

position (dx, dy) over time dt.
 Then, from 2nd assumption:
 Apply

CS4243

Taylor’s series expansion:

Motion Tracking

21

 Omit

higher order terms and divide by dt

 Denote

 Then,

 u,

v are unknown.
 2 unknowns, 1 equation: can’t solve!

CS4243

Motion Tracking

22

Any difference in motion?

CS4243

Motion Tracking

23

Any difference in motion?

CS4243

Motion Tracking

24

Any difference in motion?

CS4243

Motion Tracking

25

Any difference in motion?

CS4243

Motion Tracking

26

Any difference in motion?

CS4243

Motion Tracking

27

Any difference in motion?

CS4243

Motion Tracking

28

Aperture Problem


Homogeneous region

 Ix

= Iy = It = 0.
 No change in local region.
 Cannot detect motion.

CS4243

Motion Tracking

29

Aperture Problem


Edge

 Ix

and Iy are zero along edge
 Cannot measure motion tangential to edge
 Ix and Iy are non-zero normal to edge
 Can measure motion normal to edge
 So, cannot measure actual motion

CS4243

Motion Tracking

30

Aperture Problem


Corner

 Ix

and Iy are non-zero in two perpendicular directions
 2 unknowns, 2 equations
 Can measure actual motion

CS4243

Motion Tracking

31

Lucas-Kanade Method


Consider two consecutive image frames I and J:
I

J
x+d
d
x

x

moves from x = (x, y)T to x + d.
 d = (u, v) T
 Object

CS4243

Motion Tracking

32



So,



Or



Due to noise, there’s an error at position x:



Sum error over small window W at position x:

Similar to
template matching
CS4243

weight
Motion Tracking

33



If E is small, patterns in I and J match well.



So, find d that minimises E:
E / d = 0, compute d that minimises E.
 First, expand I(x – d) by Taylor’s series expansion:
 Set

 Omit

higher order terms:

 Write

in matrix form:

Intensity gradient
CS4243

Motion Tracking

34

 Now,

error E at position x is:

 Now,

differentiate E with respect to d (exercise):

 Setting

E / d = 0 gives

b

the only
unknown

Z

CS4243

Motion Tracking

35

 So,

2



CS4243

we get

unknowns, 2 equations. Can solve for d = (u, v).

What happen to aperture problem?
Did it disappear?
Motion Tracking

36

Lucas-Kanade + Tomasi


Lucas-Kanade algorithm is often used with
Tomasi’s feature
 Apply

Tomasi’s method to detect good features.
 Apply LK method to compute d for each pixel.
 Accept d only for good features.

CS4243

Motion Tracking

37

Example

Can you spot tracking errors?

CS4243

Motion Tracking

38

Constraints


Math of LK tracker assumes d is small.



In implementation, W is also small.



LK tracker is good only for small displacement.

CS4243

Motion Tracking

39

How to handle large displacement?


What if we scale down images?
 Displacements

CS4243

are smaller!

Motion Tracking

40

Image Pyramid
Usually, smoothen
image with Gaussian
filter before scaling
down.

downsample

scaled image

input image
CS4243

Motion Tracking

41

LK Tracking with Image Pyramid

CS4243

Motion Tracking

42



Construct image pyramids.



Apply LK tracker to low-resolution images.



Propagate results to higher-resolution images.



Apply LK tracker to higher-resolution images.

CS4243

Motion Tracking

43

Example

Tracking results are more accurate.

CS4243

Motion Tracking

44

Summary


Efficient algorithm, no explicit search.



Has aperture problem;
track good features only



LK tracker can’t track large displacement.



Use LK + image pyramid for large displacement.

CS4243

Motion Tracking

45

Software


OpenCV supports LK and LK with pyramid.



[Bir] offers LK with Tomasi’s features & pyramid.

CS4243

Motion Tracking

46

Appendix


Calculation of Ix, Iy, It
 Use

finite difference method
 Forward difference
Ix = I(x  1, y, t)  I(x, y, t)
Iy = I(x, y  1, t)  I(x, y, t)
It = I(x, y, t  1)  I(x, y, t)
 Backward difference
Ix = I(x, y, t)  I(x  1, y, t)
Iy = I(x, y, t)  I(x, y  1, t)
It = I(x, y, t)  I(x, y, t  1)

CS4243

Motion Tracking

47

Further Readings


Lucas-Kanade tracking with pyramid:
[BK08] Chapter 10.



Optical flow: [Sze10] Section 8.4.



Hierarchical motion estimation (with image
pyramid): [Sze10] Section 8.1.1.

CS4243

Motion Tracking

48

References


[Bir] S. Birchfield. KLT: An implementation of the Kanade-LucasTomasi feature tracker. http://vision.stanford.edu/~birch/klt/.



[BK08] Bradski and Kaehler. Learning OpenCV: Computer Vision
with the OpenCV Library. O’Reilly, 2008.



[LK81] B. D. Lucas and T. Kanade. An iterative image registration
technique with an application to stereo vision. In Proceedings of 7th
International Joint Conference on Artificial Intelligence, pages 674–
679, 1981.



[ST94] J. Shi and C. Tomasi. Good features to track. In Proceedings
of IEEE Conference on Computer Vision and Pattern Recognition,
pages 593–600, 1994.



[Sze10] R. Szeliski. Computer Vision: Algorithms and Applications.
Springer, 2010.

CS4243

Motion Tracking

49

References


CS4243

[TK91] C. Tomasi and T. Kanade. Detection and tracking of point
features. Technical Report CMU-CS-91-132, School of Computer
Science, Carnegie Mellon University, 1991.

Motion Tracking

50


Slide 45

Leow Wee Kheng
CS4243 Computer Vision and Pattern Recognition

Motion Tracking

CS4243

Motion Tracking

1

Changes are everywhere!

CS4243

Motion Tracking

2

Illumination change

CS4243

Motion Tracking

3

Shape change

CS4243

Motion Tracking

4

Object motion

CS4243

Motion Tracking

5

Camera motion

CS4243

Motion Tracking

6

Object & camera motion

CS4243

Motion Tracking

7

Everything changes!

CS4243

Motion Tracking

8

Motion analysis is tough in general!

We focus on object / camera motion.

CS4243

Motion Tracking

9

Change Detection


Detects any change in two video frames.



Straightforward method:
 Compute

difference between corresponding pixels:



: intensity / colour at (x, y) in frame t.
> threshold, has large difference.

 If

CS4243

Motion Tracking

10

Any difference?

No

CS4243

Motion Tracking

11

Any difference?

Yes,
illumination
change

CS4243

Motion Tracking

12

Any difference?

Yes,
position change

CS4243

Motion Tracking

13

Change Detection


Can detect
 Illumination

change
 Position change
 Illumination and position change


But, cannot distinguish between them.



Need to detect and measure position change.

CS4243

Motion Tracking

14

Motion Tracking


Two approaches
 Feature-based
 Intensity

CS4243

gradient-based

Motion Tracking

15

Feature-based Motion Tracking


Look for distinct features that change positions.

 Eagle’s

wing tips change positions.
 Tree tops don’t change positions.
CS4243

Motion Tracking

16

Basic Ideas
1.

Look for distinct features in current frame.

2.

For each feature


Search for matching feature within neighbourhood
in next frame.
 Difference in positions  displacement.
 Velocity = displacement / time difference.

CS4243

Motion Tracking

17

Basic Ideas
displacement

feature area

search area

CS4243

Motion Tracking

18

What feature to use?


Harris corner



Tomasi’s feature



Feature descriptors: SIFT, SURF, GLOH, etc.



Others

CS4243

Motion Tracking

19

Summary


Simple algorithm.



Can be slow if search area is large.



Can constrain search area with prior knowledge.

CS4243

Motion Tracking

20

Gradient-based Motion Tracking


Two basic assumptions
 Intensity

changes smoothly with position.
 Intensity of object doesn’t change over time.


Suppose an object is in motion.
 Change

position (dx, dy) over time dt.
 Then, from 2nd assumption:
 Apply

CS4243

Taylor’s series expansion:

Motion Tracking

21

 Omit

higher order terms and divide by dt

 Denote

 Then,

 u,

v are unknown.
 2 unknowns, 1 equation: can’t solve!

CS4243

Motion Tracking

22

Any difference in motion?

CS4243

Motion Tracking

23

Any difference in motion?

CS4243

Motion Tracking

24

Any difference in motion?

CS4243

Motion Tracking

25

Any difference in motion?

CS4243

Motion Tracking

26

Any difference in motion?

CS4243

Motion Tracking

27

Any difference in motion?

CS4243

Motion Tracking

28

Aperture Problem


Homogeneous region

 Ix

= Iy = It = 0.
 No change in local region.
 Cannot detect motion.

CS4243

Motion Tracking

29

Aperture Problem


Edge

 Ix

and Iy are zero along edge
 Cannot measure motion tangential to edge
 Ix and Iy are non-zero normal to edge
 Can measure motion normal to edge
 So, cannot measure actual motion

CS4243

Motion Tracking

30

Aperture Problem


Corner

 Ix

and Iy are non-zero in two perpendicular directions
 2 unknowns, 2 equations
 Can measure actual motion

CS4243

Motion Tracking

31

Lucas-Kanade Method


Consider two consecutive image frames I and J:
I

J
x+d
d
x

x

moves from x = (x, y)T to x + d.
 d = (u, v) T
 Object

CS4243

Motion Tracking

32



So,



Or



Due to noise, there’s an error at position x:



Sum error over small window W at position x:

Similar to
template matching
CS4243

weight
Motion Tracking

33



If E is small, patterns in I and J match well.



So, find d that minimises E:
E / d = 0, compute d that minimises E.
 First, expand I(x – d) by Taylor’s series expansion:
 Set

 Omit

higher order terms:

 Write

in matrix form:

Intensity gradient
CS4243

Motion Tracking

34

 Now,

error E at position x is:

 Now,

differentiate E with respect to d (exercise):

 Setting

E / d = 0 gives

b

the only
unknown

Z

CS4243

Motion Tracking

35

 So,

2



CS4243

we get

unknowns, 2 equations. Can solve for d = (u, v).

What happen to aperture problem?
Did it disappear?
Motion Tracking

36

Lucas-Kanade + Tomasi


Lucas-Kanade algorithm is often used with
Tomasi’s feature
 Apply

Tomasi’s method to detect good features.
 Apply LK method to compute d for each pixel.
 Accept d only for good features.

CS4243

Motion Tracking

37

Example

Can you spot tracking errors?

CS4243

Motion Tracking

38

Constraints


Math of LK tracker assumes d is small.



In implementation, W is also small.



LK tracker is good only for small displacement.

CS4243

Motion Tracking

39

How to handle large displacement?


What if we scale down images?
 Displacements

CS4243

are smaller!

Motion Tracking

40

Image Pyramid
Usually, smoothen
image with Gaussian
filter before scaling
down.

downsample

scaled image

input image
CS4243

Motion Tracking

41

LK Tracking with Image Pyramid

CS4243

Motion Tracking

42



Construct image pyramids.



Apply LK tracker to low-resolution images.



Propagate results to higher-resolution images.



Apply LK tracker to higher-resolution images.

CS4243

Motion Tracking

43

Example

Tracking results are more accurate.

CS4243

Motion Tracking

44

Summary


Efficient algorithm, no explicit search.



Has aperture problem;
track good features only



LK tracker can’t track large displacement.



Use LK + image pyramid for large displacement.

CS4243

Motion Tracking

45

Software


OpenCV supports LK and LK with pyramid.



[Bir] offers LK with Tomasi’s features & pyramid.

CS4243

Motion Tracking

46

Appendix


Calculation of Ix, Iy, It
 Use

finite difference method
 Forward difference
Ix = I(x  1, y, t)  I(x, y, t)
Iy = I(x, y  1, t)  I(x, y, t)
It = I(x, y, t  1)  I(x, y, t)
 Backward difference
Ix = I(x, y, t)  I(x  1, y, t)
Iy = I(x, y, t)  I(x, y  1, t)
It = I(x, y, t)  I(x, y, t  1)

CS4243

Motion Tracking

47

Further Readings


Lucas-Kanade tracking with pyramid:
[BK08] Chapter 10.



Optical flow: [Sze10] Section 8.4.



Hierarchical motion estimation (with image
pyramid): [Sze10] Section 8.1.1.

CS4243

Motion Tracking

48

References


[Bir] S. Birchfield. KLT: An implementation of the Kanade-LucasTomasi feature tracker. http://vision.stanford.edu/~birch/klt/.



[BK08] Bradski and Kaehler. Learning OpenCV: Computer Vision
with the OpenCV Library. O’Reilly, 2008.



[LK81] B. D. Lucas and T. Kanade. An iterative image registration
technique with an application to stereo vision. In Proceedings of 7th
International Joint Conference on Artificial Intelligence, pages 674–
679, 1981.



[ST94] J. Shi and C. Tomasi. Good features to track. In Proceedings
of IEEE Conference on Computer Vision and Pattern Recognition,
pages 593–600, 1994.



[Sze10] R. Szeliski. Computer Vision: Algorithms and Applications.
Springer, 2010.

CS4243

Motion Tracking

49

References


CS4243

[TK91] C. Tomasi and T. Kanade. Detection and tracking of point
features. Technical Report CMU-CS-91-132, School of Computer
Science, Carnegie Mellon University, 1991.

Motion Tracking

50


Slide 46

Leow Wee Kheng
CS4243 Computer Vision and Pattern Recognition

Motion Tracking

CS4243

Motion Tracking

1

Changes are everywhere!

CS4243

Motion Tracking

2

Illumination change

CS4243

Motion Tracking

3

Shape change

CS4243

Motion Tracking

4

Object motion

CS4243

Motion Tracking

5

Camera motion

CS4243

Motion Tracking

6

Object & camera motion

CS4243

Motion Tracking

7

Everything changes!

CS4243

Motion Tracking

8

Motion analysis is tough in general!

We focus on object / camera motion.

CS4243

Motion Tracking

9

Change Detection


Detects any change in two video frames.



Straightforward method:
 Compute

difference between corresponding pixels:



: intensity / colour at (x, y) in frame t.
> threshold, has large difference.

 If

CS4243

Motion Tracking

10

Any difference?

No

CS4243

Motion Tracking

11

Any difference?

Yes,
illumination
change

CS4243

Motion Tracking

12

Any difference?

Yes,
position change

CS4243

Motion Tracking

13

Change Detection


Can detect
 Illumination

change
 Position change
 Illumination and position change


But, cannot distinguish between them.



Need to detect and measure position change.

CS4243

Motion Tracking

14

Motion Tracking


Two approaches
 Feature-based
 Intensity

CS4243

gradient-based

Motion Tracking

15

Feature-based Motion Tracking


Look for distinct features that change positions.

 Eagle’s

wing tips change positions.
 Tree tops don’t change positions.
CS4243

Motion Tracking

16

Basic Ideas
1.

Look for distinct features in current frame.

2.

For each feature


Search for matching feature within neighbourhood
in next frame.
 Difference in positions  displacement.
 Velocity = displacement / time difference.

CS4243

Motion Tracking

17

Basic Ideas
displacement

feature area

search area

CS4243

Motion Tracking

18

What feature to use?


Harris corner



Tomasi’s feature



Feature descriptors: SIFT, SURF, GLOH, etc.



Others

CS4243

Motion Tracking

19

Summary


Simple algorithm.



Can be slow if search area is large.



Can constrain search area with prior knowledge.

CS4243

Motion Tracking

20

Gradient-based Motion Tracking


Two basic assumptions
 Intensity

changes smoothly with position.
 Intensity of object doesn’t change over time.


Suppose an object is in motion.
 Change

position (dx, dy) over time dt.
 Then, from 2nd assumption:
 Apply

CS4243

Taylor’s series expansion:

Motion Tracking

21

 Omit

higher order terms and divide by dt

 Denote

 Then,

 u,

v are unknown.
 2 unknowns, 1 equation: can’t solve!

CS4243

Motion Tracking

22

Any difference in motion?

CS4243

Motion Tracking

23

Any difference in motion?

CS4243

Motion Tracking

24

Any difference in motion?

CS4243

Motion Tracking

25

Any difference in motion?

CS4243

Motion Tracking

26

Any difference in motion?

CS4243

Motion Tracking

27

Any difference in motion?

CS4243

Motion Tracking

28

Aperture Problem


Homogeneous region

 Ix

= Iy = It = 0.
 No change in local region.
 Cannot detect motion.

CS4243

Motion Tracking

29

Aperture Problem


Edge

 Ix

and Iy are zero along edge
 Cannot measure motion tangential to edge
 Ix and Iy are non-zero normal to edge
 Can measure motion normal to edge
 So, cannot measure actual motion

CS4243

Motion Tracking

30

Aperture Problem


Corner

 Ix

and Iy are non-zero in two perpendicular directions
 2 unknowns, 2 equations
 Can measure actual motion

CS4243

Motion Tracking

31

Lucas-Kanade Method


Consider two consecutive image frames I and J:
I

J
x+d
d
x

x

moves from x = (x, y)T to x + d.
 d = (u, v) T
 Object

CS4243

Motion Tracking

32



So,



Or



Due to noise, there’s an error at position x:



Sum error over small window W at position x:

Similar to
template matching
CS4243

weight
Motion Tracking

33



If E is small, patterns in I and J match well.



So, find d that minimises E:
E / d = 0, compute d that minimises E.
 First, expand I(x – d) by Taylor’s series expansion:
 Set

 Omit

higher order terms:

 Write

in matrix form:

Intensity gradient
CS4243

Motion Tracking

34

 Now,

error E at position x is:

 Now,

differentiate E with respect to d (exercise):

 Setting

E / d = 0 gives

b

the only
unknown

Z

CS4243

Motion Tracking

35

 So,

2



CS4243

we get

unknowns, 2 equations. Can solve for d = (u, v).

What happen to aperture problem?
Did it disappear?
Motion Tracking

36

Lucas-Kanade + Tomasi


Lucas-Kanade algorithm is often used with
Tomasi’s feature
 Apply

Tomasi’s method to detect good features.
 Apply LK method to compute d for each pixel.
 Accept d only for good features.

CS4243

Motion Tracking

37

Example

Can you spot tracking errors?

CS4243

Motion Tracking

38

Constraints


Math of LK tracker assumes d is small.



In implementation, W is also small.



LK tracker is good only for small displacement.

CS4243

Motion Tracking

39

How to handle large displacement?


What if we scale down images?
 Displacements

CS4243

are smaller!

Motion Tracking

40

Image Pyramid
Usually, smoothen
image with Gaussian
filter before scaling
down.

downsample

scaled image

input image
CS4243

Motion Tracking

41

LK Tracking with Image Pyramid

CS4243

Motion Tracking

42



Construct image pyramids.



Apply LK tracker to low-resolution images.



Propagate results to higher-resolution images.



Apply LK tracker to higher-resolution images.

CS4243

Motion Tracking

43

Example

Tracking results are more accurate.

CS4243

Motion Tracking

44

Summary


Efficient algorithm, no explicit search.



Has aperture problem;
track good features only



LK tracker can’t track large displacement.



Use LK + image pyramid for large displacement.

CS4243

Motion Tracking

45

Software


OpenCV supports LK and LK with pyramid.



[Bir] offers LK with Tomasi’s features & pyramid.

CS4243

Motion Tracking

46

Appendix


Calculation of Ix, Iy, It
 Use

finite difference method
 Forward difference
Ix = I(x  1, y, t)  I(x, y, t)
Iy = I(x, y  1, t)  I(x, y, t)
It = I(x, y, t  1)  I(x, y, t)
 Backward difference
Ix = I(x, y, t)  I(x  1, y, t)
Iy = I(x, y, t)  I(x, y  1, t)
It = I(x, y, t)  I(x, y, t  1)

CS4243

Motion Tracking

47

Further Readings


Lucas-Kanade tracking with pyramid:
[BK08] Chapter 10.



Optical flow: [Sze10] Section 8.4.



Hierarchical motion estimation (with image
pyramid): [Sze10] Section 8.1.1.

CS4243

Motion Tracking

48

References


[Bir] S. Birchfield. KLT: An implementation of the Kanade-LucasTomasi feature tracker. http://vision.stanford.edu/~birch/klt/.



[BK08] Bradski and Kaehler. Learning OpenCV: Computer Vision
with the OpenCV Library. O’Reilly, 2008.



[LK81] B. D. Lucas and T. Kanade. An iterative image registration
technique with an application to stereo vision. In Proceedings of 7th
International Joint Conference on Artificial Intelligence, pages 674–
679, 1981.



[ST94] J. Shi and C. Tomasi. Good features to track. In Proceedings
of IEEE Conference on Computer Vision and Pattern Recognition,
pages 593–600, 1994.



[Sze10] R. Szeliski. Computer Vision: Algorithms and Applications.
Springer, 2010.

CS4243

Motion Tracking

49

References


CS4243

[TK91] C. Tomasi and T. Kanade. Detection and tracking of point
features. Technical Report CMU-CS-91-132, School of Computer
Science, Carnegie Mellon University, 1991.

Motion Tracking

50


Slide 47

Leow Wee Kheng
CS4243 Computer Vision and Pattern Recognition

Motion Tracking

CS4243

Motion Tracking

1

Changes are everywhere!

CS4243

Motion Tracking

2

Illumination change

CS4243

Motion Tracking

3

Shape change

CS4243

Motion Tracking

4

Object motion

CS4243

Motion Tracking

5

Camera motion

CS4243

Motion Tracking

6

Object & camera motion

CS4243

Motion Tracking

7

Everything changes!

CS4243

Motion Tracking

8

Motion analysis is tough in general!

We focus on object / camera motion.

CS4243

Motion Tracking

9

Change Detection


Detects any change in two video frames.



Straightforward method:
 Compute

difference between corresponding pixels:



: intensity / colour at (x, y) in frame t.
> threshold, has large difference.

 If

CS4243

Motion Tracking

10

Any difference?

No

CS4243

Motion Tracking

11

Any difference?

Yes,
illumination
change

CS4243

Motion Tracking

12

Any difference?

Yes,
position change

CS4243

Motion Tracking

13

Change Detection


Can detect
 Illumination

change
 Position change
 Illumination and position change


But, cannot distinguish between them.



Need to detect and measure position change.

CS4243

Motion Tracking

14

Motion Tracking


Two approaches
 Feature-based
 Intensity

CS4243

gradient-based

Motion Tracking

15

Feature-based Motion Tracking


Look for distinct features that change positions.

 Eagle’s

wing tips change positions.
 Tree tops don’t change positions.
CS4243

Motion Tracking

16

Basic Ideas
1.

Look for distinct features in current frame.

2.

For each feature


Search for matching feature within neighbourhood
in next frame.
 Difference in positions  displacement.
 Velocity = displacement / time difference.

CS4243

Motion Tracking

17

Basic Ideas
displacement

feature area

search area

CS4243

Motion Tracking

18

What feature to use?


Harris corner



Tomasi’s feature



Feature descriptors: SIFT, SURF, GLOH, etc.



Others

CS4243

Motion Tracking

19

Summary


Simple algorithm.



Can be slow if search area is large.



Can constrain search area with prior knowledge.

CS4243

Motion Tracking

20

Gradient-based Motion Tracking


Two basic assumptions
 Intensity

changes smoothly with position.
 Intensity of object doesn’t change over time.


Suppose an object is in motion.
 Change

position (dx, dy) over time dt.
 Then, from 2nd assumption:
 Apply

CS4243

Taylor’s series expansion:

Motion Tracking

21

 Omit

higher order terms and divide by dt

 Denote

 Then,

 u,

v are unknown.
 2 unknowns, 1 equation: can’t solve!

CS4243

Motion Tracking

22

Any difference in motion?

CS4243

Motion Tracking

23

Any difference in motion?

CS4243

Motion Tracking

24

Any difference in motion?

CS4243

Motion Tracking

25

Any difference in motion?

CS4243

Motion Tracking

26

Any difference in motion?

CS4243

Motion Tracking

27

Any difference in motion?

CS4243

Motion Tracking

28

Aperture Problem


Homogeneous region

 Ix

= Iy = It = 0.
 No change in local region.
 Cannot detect motion.

CS4243

Motion Tracking

29

Aperture Problem


Edge

 Ix

and Iy are zero along edge
 Cannot measure motion tangential to edge
 Ix and Iy are non-zero normal to edge
 Can measure motion normal to edge
 So, cannot measure actual motion

CS4243

Motion Tracking

30

Aperture Problem


Corner

 Ix

and Iy are non-zero in two perpendicular directions
 2 unknowns, 2 equations
 Can measure actual motion

CS4243

Motion Tracking

31

Lucas-Kanade Method


Consider two consecutive image frames I and J:
I

J
x+d
d
x

x

moves from x = (x, y)T to x + d.
 d = (u, v) T
 Object

CS4243

Motion Tracking

32



So,



Or



Due to noise, there’s an error at position x:



Sum error over small window W at position x:

Similar to
template matching
CS4243

weight
Motion Tracking

33



If E is small, patterns in I and J match well.



So, find d that minimises E:
E / d = 0, compute d that minimises E.
 First, expand I(x – d) by Taylor’s series expansion:
 Set

 Omit

higher order terms:

 Write

in matrix form:

Intensity gradient
CS4243

Motion Tracking

34

 Now,

error E at position x is:

 Now,

differentiate E with respect to d (exercise):

 Setting

E / d = 0 gives

b

the only
unknown

Z

CS4243

Motion Tracking

35

 So,

2



CS4243

we get

unknowns, 2 equations. Can solve for d = (u, v).

What happen to aperture problem?
Did it disappear?
Motion Tracking

36

Lucas-Kanade + Tomasi


Lucas-Kanade algorithm is often used with
Tomasi’s feature
 Apply

Tomasi’s method to detect good features.
 Apply LK method to compute d for each pixel.
 Accept d only for good features.

CS4243

Motion Tracking

37

Example

Can you spot tracking errors?

CS4243

Motion Tracking

38

Constraints


Math of LK tracker assumes d is small.



In implementation, W is also small.



LK tracker is good only for small displacement.

CS4243

Motion Tracking

39

How to handle large displacement?


What if we scale down images?
 Displacements

CS4243

are smaller!

Motion Tracking

40

Image Pyramid
Usually, smoothen
image with Gaussian
filter before scaling
down.

downsample

scaled image

input image
CS4243

Motion Tracking

41

LK Tracking with Image Pyramid

CS4243

Motion Tracking

42



Construct image pyramids.



Apply LK tracker to low-resolution images.



Propagate results to higher-resolution images.



Apply LK tracker to higher-resolution images.

CS4243

Motion Tracking

43

Example

Tracking results are more accurate.

CS4243

Motion Tracking

44

Summary


Efficient algorithm, no explicit search.



Has aperture problem;
track good features only



LK tracker can’t track large displacement.



Use LK + image pyramid for large displacement.

CS4243

Motion Tracking

45

Software


OpenCV supports LK and LK with pyramid.



[Bir] offers LK with Tomasi’s features & pyramid.

CS4243

Motion Tracking

46

Appendix


Calculation of Ix, Iy, It
 Use

finite difference method
 Forward difference
Ix = I(x  1, y, t)  I(x, y, t)
Iy = I(x, y  1, t)  I(x, y, t)
It = I(x, y, t  1)  I(x, y, t)
 Backward difference
Ix = I(x, y, t)  I(x  1, y, t)
Iy = I(x, y, t)  I(x, y  1, t)
It = I(x, y, t)  I(x, y, t  1)

CS4243

Motion Tracking

47

Further Readings


Lucas-Kanade tracking with pyramid:
[BK08] Chapter 10.



Optical flow: [Sze10] Section 8.4.



Hierarchical motion estimation (with image
pyramid): [Sze10] Section 8.1.1.

CS4243

Motion Tracking

48

References


[Bir] S. Birchfield. KLT: An implementation of the Kanade-LucasTomasi feature tracker. http://vision.stanford.edu/~birch/klt/.



[BK08] Bradski and Kaehler. Learning OpenCV: Computer Vision
with the OpenCV Library. O’Reilly, 2008.



[LK81] B. D. Lucas and T. Kanade. An iterative image registration
technique with an application to stereo vision. In Proceedings of 7th
International Joint Conference on Artificial Intelligence, pages 674–
679, 1981.



[ST94] J. Shi and C. Tomasi. Good features to track. In Proceedings
of IEEE Conference on Computer Vision and Pattern Recognition,
pages 593–600, 1994.



[Sze10] R. Szeliski. Computer Vision: Algorithms and Applications.
Springer, 2010.

CS4243

Motion Tracking

49

References


CS4243

[TK91] C. Tomasi and T. Kanade. Detection and tracking of point
features. Technical Report CMU-CS-91-132, School of Computer
Science, Carnegie Mellon University, 1991.

Motion Tracking

50


Slide 48

Leow Wee Kheng
CS4243 Computer Vision and Pattern Recognition

Motion Tracking

CS4243

Motion Tracking

1

Changes are everywhere!

CS4243

Motion Tracking

2

Illumination change

CS4243

Motion Tracking

3

Shape change

CS4243

Motion Tracking

4

Object motion

CS4243

Motion Tracking

5

Camera motion

CS4243

Motion Tracking

6

Object & camera motion

CS4243

Motion Tracking

7

Everything changes!

CS4243

Motion Tracking

8

Motion analysis is tough in general!

We focus on object / camera motion.

CS4243

Motion Tracking

9

Change Detection


Detects any change in two video frames.



Straightforward method:
 Compute

difference between corresponding pixels:



: intensity / colour at (x, y) in frame t.
> threshold, has large difference.

 If

CS4243

Motion Tracking

10

Any difference?

No

CS4243

Motion Tracking

11

Any difference?

Yes,
illumination
change

CS4243

Motion Tracking

12

Any difference?

Yes,
position change

CS4243

Motion Tracking

13

Change Detection


Can detect
 Illumination

change
 Position change
 Illumination and position change


But, cannot distinguish between them.



Need to detect and measure position change.

CS4243

Motion Tracking

14

Motion Tracking


Two approaches
 Feature-based
 Intensity

CS4243

gradient-based

Motion Tracking

15

Feature-based Motion Tracking


Look for distinct features that change positions.

 Eagle’s

wing tips change positions.
 Tree tops don’t change positions.
CS4243

Motion Tracking

16

Basic Ideas
1.

Look for distinct features in current frame.

2.

For each feature


Search for matching feature within neighbourhood
in next frame.
 Difference in positions  displacement.
 Velocity = displacement / time difference.

CS4243

Motion Tracking

17

Basic Ideas
displacement

feature area

search area

CS4243

Motion Tracking

18

What feature to use?


Harris corner



Tomasi’s feature



Feature descriptors: SIFT, SURF, GLOH, etc.



Others

CS4243

Motion Tracking

19

Summary


Simple algorithm.



Can be slow if search area is large.



Can constrain search area with prior knowledge.

CS4243

Motion Tracking

20

Gradient-based Motion Tracking


Two basic assumptions
 Intensity

changes smoothly with position.
 Intensity of object doesn’t change over time.


Suppose an object is in motion.
 Change

position (dx, dy) over time dt.
 Then, from 2nd assumption:
 Apply

CS4243

Taylor’s series expansion:

Motion Tracking

21

 Omit

higher order terms and divide by dt

 Denote

 Then,

 u,

v are unknown.
 2 unknowns, 1 equation: can’t solve!

CS4243

Motion Tracking

22

Any difference in motion?

CS4243

Motion Tracking

23

Any difference in motion?

CS4243

Motion Tracking

24

Any difference in motion?

CS4243

Motion Tracking

25

Any difference in motion?

CS4243

Motion Tracking

26

Any difference in motion?

CS4243

Motion Tracking

27

Any difference in motion?

CS4243

Motion Tracking

28

Aperture Problem


Homogeneous region

 Ix

= Iy = It = 0.
 No change in local region.
 Cannot detect motion.

CS4243

Motion Tracking

29

Aperture Problem


Edge

 Ix

and Iy are zero along edge
 Cannot measure motion tangential to edge
 Ix and Iy are non-zero normal to edge
 Can measure motion normal to edge
 So, cannot measure actual motion

CS4243

Motion Tracking

30

Aperture Problem


Corner

 Ix

and Iy are non-zero in two perpendicular directions
 2 unknowns, 2 equations
 Can measure actual motion

CS4243

Motion Tracking

31

Lucas-Kanade Method


Consider two consecutive image frames I and J:
I

J
x+d
d
x

x

moves from x = (x, y)T to x + d.
 d = (u, v) T
 Object

CS4243

Motion Tracking

32



So,



Or



Due to noise, there’s an error at position x:



Sum error over small window W at position x:

Similar to
template matching
CS4243

weight
Motion Tracking

33



If E is small, patterns in I and J match well.



So, find d that minimises E:
E / d = 0, compute d that minimises E.
 First, expand I(x – d) by Taylor’s series expansion:
 Set

 Omit

higher order terms:

 Write

in matrix form:

Intensity gradient
CS4243

Motion Tracking

34

 Now,

error E at position x is:

 Now,

differentiate E with respect to d (exercise):

 Setting

E / d = 0 gives

b

the only
unknown

Z

CS4243

Motion Tracking

35

 So,

2



CS4243

we get

unknowns, 2 equations. Can solve for d = (u, v).

What happen to aperture problem?
Did it disappear?
Motion Tracking

36

Lucas-Kanade + Tomasi


Lucas-Kanade algorithm is often used with
Tomasi’s feature
 Apply

Tomasi’s method to detect good features.
 Apply LK method to compute d for each pixel.
 Accept d only for good features.

CS4243

Motion Tracking

37

Example

Can you spot tracking errors?

CS4243

Motion Tracking

38

Constraints


Math of LK tracker assumes d is small.



In implementation, W is also small.



LK tracker is good only for small displacement.

CS4243

Motion Tracking

39

How to handle large displacement?


What if we scale down images?
 Displacements

CS4243

are smaller!

Motion Tracking

40

Image Pyramid
Usually, smoothen
image with Gaussian
filter before scaling
down.

downsample

scaled image

input image
CS4243

Motion Tracking

41

LK Tracking with Image Pyramid

CS4243

Motion Tracking

42



Construct image pyramids.



Apply LK tracker to low-resolution images.



Propagate results to higher-resolution images.



Apply LK tracker to higher-resolution images.

CS4243

Motion Tracking

43

Example

Tracking results are more accurate.

CS4243

Motion Tracking

44

Summary


Efficient algorithm, no explicit search.



Has aperture problem;
track good features only



LK tracker can’t track large displacement.



Use LK + image pyramid for large displacement.

CS4243

Motion Tracking

45

Software


OpenCV supports LK and LK with pyramid.



[Bir] offers LK with Tomasi’s features & pyramid.

CS4243

Motion Tracking

46

Appendix


Calculation of Ix, Iy, It
 Use

finite difference method
 Forward difference
Ix = I(x  1, y, t)  I(x, y, t)
Iy = I(x, y  1, t)  I(x, y, t)
It = I(x, y, t  1)  I(x, y, t)
 Backward difference
Ix = I(x, y, t)  I(x  1, y, t)
Iy = I(x, y, t)  I(x, y  1, t)
It = I(x, y, t)  I(x, y, t  1)

CS4243

Motion Tracking

47

Further Readings


Lucas-Kanade tracking with pyramid:
[BK08] Chapter 10.



Optical flow: [Sze10] Section 8.4.



Hierarchical motion estimation (with image
pyramid): [Sze10] Section 8.1.1.

CS4243

Motion Tracking

48

References


[Bir] S. Birchfield. KLT: An implementation of the Kanade-LucasTomasi feature tracker. http://vision.stanford.edu/~birch/klt/.



[BK08] Bradski and Kaehler. Learning OpenCV: Computer Vision
with the OpenCV Library. O’Reilly, 2008.



[LK81] B. D. Lucas and T. Kanade. An iterative image registration
technique with an application to stereo vision. In Proceedings of 7th
International Joint Conference on Artificial Intelligence, pages 674–
679, 1981.



[ST94] J. Shi and C. Tomasi. Good features to track. In Proceedings
of IEEE Conference on Computer Vision and Pattern Recognition,
pages 593–600, 1994.



[Sze10] R. Szeliski. Computer Vision: Algorithms and Applications.
Springer, 2010.

CS4243

Motion Tracking

49

References


CS4243

[TK91] C. Tomasi and T. Kanade. Detection and tracking of point
features. Technical Report CMU-CS-91-132, School of Computer
Science, Carnegie Mellon University, 1991.

Motion Tracking

50


Slide 49

Leow Wee Kheng
CS4243 Computer Vision and Pattern Recognition

Motion Tracking

CS4243

Motion Tracking

1

Changes are everywhere!

CS4243

Motion Tracking

2

Illumination change

CS4243

Motion Tracking

3

Shape change

CS4243

Motion Tracking

4

Object motion

CS4243

Motion Tracking

5

Camera motion

CS4243

Motion Tracking

6

Object & camera motion

CS4243

Motion Tracking

7

Everything changes!

CS4243

Motion Tracking

8

Motion analysis is tough in general!

We focus on object / camera motion.

CS4243

Motion Tracking

9

Change Detection


Detects any change in two video frames.



Straightforward method:
 Compute

difference between corresponding pixels:



: intensity / colour at (x, y) in frame t.
> threshold, has large difference.

 If

CS4243

Motion Tracking

10

Any difference?

No

CS4243

Motion Tracking

11

Any difference?

Yes,
illumination
change

CS4243

Motion Tracking

12

Any difference?

Yes,
position change

CS4243

Motion Tracking

13

Change Detection


Can detect
 Illumination

change
 Position change
 Illumination and position change


But, cannot distinguish between them.



Need to detect and measure position change.

CS4243

Motion Tracking

14

Motion Tracking


Two approaches
 Feature-based
 Intensity

CS4243

gradient-based

Motion Tracking

15

Feature-based Motion Tracking


Look for distinct features that change positions.

 Eagle’s

wing tips change positions.
 Tree tops don’t change positions.
CS4243

Motion Tracking

16

Basic Ideas
1.

Look for distinct features in current frame.

2.

For each feature


Search for matching feature within neighbourhood
in next frame.
 Difference in positions  displacement.
 Velocity = displacement / time difference.

CS4243

Motion Tracking

17

Basic Ideas
displacement

feature area

search area

CS4243

Motion Tracking

18

What feature to use?


Harris corner



Tomasi’s feature



Feature descriptors: SIFT, SURF, GLOH, etc.



Others

CS4243

Motion Tracking

19

Summary


Simple algorithm.



Can be slow if search area is large.



Can constrain search area with prior knowledge.

CS4243

Motion Tracking

20

Gradient-based Motion Tracking


Two basic assumptions
 Intensity

changes smoothly with position.
 Intensity of object doesn’t change over time.


Suppose an object is in motion.
 Change

position (dx, dy) over time dt.
 Then, from 2nd assumption:
 Apply

CS4243

Taylor’s series expansion:

Motion Tracking

21

 Omit

higher order terms and divide by dt

 Denote

 Then,

 u,

v are unknown.
 2 unknowns, 1 equation: can’t solve!

CS4243

Motion Tracking

22

Any difference in motion?

CS4243

Motion Tracking

23

Any difference in motion?

CS4243

Motion Tracking

24

Any difference in motion?

CS4243

Motion Tracking

25

Any difference in motion?

CS4243

Motion Tracking

26

Any difference in motion?

CS4243

Motion Tracking

27

Any difference in motion?

CS4243

Motion Tracking

28

Aperture Problem


Homogeneous region

 Ix

= Iy = It = 0.
 No change in local region.
 Cannot detect motion.

CS4243

Motion Tracking

29

Aperture Problem


Edge

 Ix

and Iy are zero along edge
 Cannot measure motion tangential to edge
 Ix and Iy are non-zero normal to edge
 Can measure motion normal to edge
 So, cannot measure actual motion

CS4243

Motion Tracking

30

Aperture Problem


Corner

 Ix

and Iy are non-zero in two perpendicular directions
 2 unknowns, 2 equations
 Can measure actual motion

CS4243

Motion Tracking

31

Lucas-Kanade Method


Consider two consecutive image frames I and J:
I

J
x+d
d
x

x

moves from x = (x, y)T to x + d.
 d = (u, v) T
 Object

CS4243

Motion Tracking

32



So,



Or



Due to noise, there’s an error at position x:



Sum error over small window W at position x:

Similar to
template matching
CS4243

weight
Motion Tracking

33



If E is small, patterns in I and J match well.



So, find d that minimises E:
E / d = 0, compute d that minimises E.
 First, expand I(x – d) by Taylor’s series expansion:
 Set

 Omit

higher order terms:

 Write

in matrix form:

Intensity gradient
CS4243

Motion Tracking

34

 Now,

error E at position x is:

 Now,

differentiate E with respect to d (exercise):

 Setting

E / d = 0 gives

b

the only
unknown

Z

CS4243

Motion Tracking

35

 So,

2



CS4243

we get

unknowns, 2 equations. Can solve for d = (u, v).

What happen to aperture problem?
Did it disappear?
Motion Tracking

36

Lucas-Kanade + Tomasi


Lucas-Kanade algorithm is often used with
Tomasi’s feature
 Apply

Tomasi’s method to detect good features.
 Apply LK method to compute d for each pixel.
 Accept d only for good features.

CS4243

Motion Tracking

37

Example

Can you spot tracking errors?

CS4243

Motion Tracking

38

Constraints


Math of LK tracker assumes d is small.



In implementation, W is also small.



LK tracker is good only for small displacement.

CS4243

Motion Tracking

39

How to handle large displacement?


What if we scale down images?
 Displacements

CS4243

are smaller!

Motion Tracking

40

Image Pyramid
Usually, smoothen
image with Gaussian
filter before scaling
down.

downsample

scaled image

input image
CS4243

Motion Tracking

41

LK Tracking with Image Pyramid

CS4243

Motion Tracking

42



Construct image pyramids.



Apply LK tracker to low-resolution images.



Propagate results to higher-resolution images.



Apply LK tracker to higher-resolution images.

CS4243

Motion Tracking

43

Example

Tracking results are more accurate.

CS4243

Motion Tracking

44

Summary


Efficient algorithm, no explicit search.



Has aperture problem;
track good features only



LK tracker can’t track large displacement.



Use LK + image pyramid for large displacement.

CS4243

Motion Tracking

45

Software


OpenCV supports LK and LK with pyramid.



[Bir] offers LK with Tomasi’s features & pyramid.

CS4243

Motion Tracking

46

Appendix


Calculation of Ix, Iy, It
 Use

finite difference method
 Forward difference
Ix = I(x  1, y, t)  I(x, y, t)
Iy = I(x, y  1, t)  I(x, y, t)
It = I(x, y, t  1)  I(x, y, t)
 Backward difference
Ix = I(x, y, t)  I(x  1, y, t)
Iy = I(x, y, t)  I(x, y  1, t)
It = I(x, y, t)  I(x, y, t  1)

CS4243

Motion Tracking

47

Further Readings


Lucas-Kanade tracking with pyramid:
[BK08] Chapter 10.



Optical flow: [Sze10] Section 8.4.



Hierarchical motion estimation (with image
pyramid): [Sze10] Section 8.1.1.

CS4243

Motion Tracking

48

References


[Bir] S. Birchfield. KLT: An implementation of the Kanade-LucasTomasi feature tracker. http://vision.stanford.edu/~birch/klt/.



[BK08] Bradski and Kaehler. Learning OpenCV: Computer Vision
with the OpenCV Library. O’Reilly, 2008.



[LK81] B. D. Lucas and T. Kanade. An iterative image registration
technique with an application to stereo vision. In Proceedings of 7th
International Joint Conference on Artificial Intelligence, pages 674–
679, 1981.



[ST94] J. Shi and C. Tomasi. Good features to track. In Proceedings
of IEEE Conference on Computer Vision and Pattern Recognition,
pages 593–600, 1994.



[Sze10] R. Szeliski. Computer Vision: Algorithms and Applications.
Springer, 2010.

CS4243

Motion Tracking

49

References


CS4243

[TK91] C. Tomasi and T. Kanade. Detection and tracking of point
features. Technical Report CMU-CS-91-132, School of Computer
Science, Carnegie Mellon University, 1991.

Motion Tracking

50


Slide 50

Leow Wee Kheng
CS4243 Computer Vision and Pattern Recognition

Motion Tracking

CS4243

Motion Tracking

1

Changes are everywhere!

CS4243

Motion Tracking

2

Illumination change

CS4243

Motion Tracking

3

Shape change

CS4243

Motion Tracking

4

Object motion

CS4243

Motion Tracking

5

Camera motion

CS4243

Motion Tracking

6

Object & camera motion

CS4243

Motion Tracking

7

Everything changes!

CS4243

Motion Tracking

8

Motion analysis is tough in general!

We focus on object / camera motion.

CS4243

Motion Tracking

9

Change Detection


Detects any change in two video frames.



Straightforward method:
 Compute

difference between corresponding pixels:



: intensity / colour at (x, y) in frame t.
> threshold, has large difference.

 If

CS4243

Motion Tracking

10

Any difference?

No

CS4243

Motion Tracking

11

Any difference?

Yes,
illumination
change

CS4243

Motion Tracking

12

Any difference?

Yes,
position change

CS4243

Motion Tracking

13

Change Detection


Can detect
 Illumination

change
 Position change
 Illumination and position change


But, cannot distinguish between them.



Need to detect and measure position change.

CS4243

Motion Tracking

14

Motion Tracking


Two approaches
 Feature-based
 Intensity

CS4243

gradient-based

Motion Tracking

15

Feature-based Motion Tracking


Look for distinct features that change positions.

 Eagle’s

wing tips change positions.
 Tree tops don’t change positions.
CS4243

Motion Tracking

16

Basic Ideas
1.

Look for distinct features in current frame.

2.

For each feature


Search for matching feature within neighbourhood
in next frame.
 Difference in positions  displacement.
 Velocity = displacement / time difference.

CS4243

Motion Tracking

17

Basic Ideas
displacement

feature area

search area

CS4243

Motion Tracking

18

What feature to use?


Harris corner



Tomasi’s feature



Feature descriptors: SIFT, SURF, GLOH, etc.



Others

CS4243

Motion Tracking

19

Summary


Simple algorithm.



Can be slow if search area is large.



Can constrain search area with prior knowledge.

CS4243

Motion Tracking

20

Gradient-based Motion Tracking


Two basic assumptions
 Intensity

changes smoothly with position.
 Intensity of object doesn’t change over time.


Suppose an object is in motion.
 Change

position (dx, dy) over time dt.
 Then, from 2nd assumption:
 Apply

CS4243

Taylor’s series expansion:

Motion Tracking

21

 Omit

higher order terms and divide by dt

 Denote

 Then,

 u,

v are unknown.
 2 unknowns, 1 equation: can’t solve!

CS4243

Motion Tracking

22

Any difference in motion?

CS4243

Motion Tracking

23

Any difference in motion?

CS4243

Motion Tracking

24

Any difference in motion?

CS4243

Motion Tracking

25

Any difference in motion?

CS4243

Motion Tracking

26

Any difference in motion?

CS4243

Motion Tracking

27

Any difference in motion?

CS4243

Motion Tracking

28

Aperture Problem


Homogeneous region

 Ix

= Iy = It = 0.
 No change in local region.
 Cannot detect motion.

CS4243

Motion Tracking

29

Aperture Problem


Edge

 Ix

and Iy are zero along edge
 Cannot measure motion tangential to edge
 Ix and Iy are non-zero normal to edge
 Can measure motion normal to edge
 So, cannot measure actual motion

CS4243

Motion Tracking

30

Aperture Problem


Corner

 Ix

and Iy are non-zero in two perpendicular directions
 2 unknowns, 2 equations
 Can measure actual motion

CS4243

Motion Tracking

31

Lucas-Kanade Method


Consider two consecutive image frames I and J:
I

J
x+d
d
x

x

moves from x = (x, y)T to x + d.
 d = (u, v) T
 Object

CS4243

Motion Tracking

32



So,



Or



Due to noise, there’s an error at position x:



Sum error over small window W at position x:

Similar to
template matching
CS4243

weight
Motion Tracking

33



If E is small, patterns in I and J match well.



So, find d that minimises E:
E / d = 0, compute d that minimises E.
 First, expand I(x – d) by Taylor’s series expansion:
 Set

 Omit

higher order terms:

 Write

in matrix form:

Intensity gradient
CS4243

Motion Tracking

34

 Now,

error E at position x is:

 Now,

differentiate E with respect to d (exercise):

 Setting

E / d = 0 gives

b

the only
unknown

Z

CS4243

Motion Tracking

35

 So,

2



CS4243

we get

unknowns, 2 equations. Can solve for d = (u, v).

What happen to aperture problem?
Did it disappear?
Motion Tracking

36

Lucas-Kanade + Tomasi


Lucas-Kanade algorithm is often used with
Tomasi’s feature
 Apply

Tomasi’s method to detect good features.
 Apply LK method to compute d for each pixel.
 Accept d only for good features.

CS4243

Motion Tracking

37

Example

Can you spot tracking errors?

CS4243

Motion Tracking

38

Constraints


Math of LK tracker assumes d is small.



In implementation, W is also small.



LK tracker is good only for small displacement.

CS4243

Motion Tracking

39

How to handle large displacement?


What if we scale down images?
 Displacements

CS4243

are smaller!

Motion Tracking

40

Image Pyramid
Usually, smoothen
image with Gaussian
filter before scaling
down.

downsample

scaled image

input image
CS4243

Motion Tracking

41

LK Tracking with Image Pyramid

CS4243

Motion Tracking

42



Construct image pyramids.



Apply LK tracker to low-resolution images.



Propagate results to higher-resolution images.



Apply LK tracker to higher-resolution images.

CS4243

Motion Tracking

43

Example

Tracking results are more accurate.

CS4243

Motion Tracking

44

Summary


Efficient algorithm, no explicit search.



Has aperture problem;
track good features only



LK tracker can’t track large displacement.



Use LK + image pyramid for large displacement.

CS4243

Motion Tracking

45

Software


OpenCV supports LK and LK with pyramid.



[Bir] offers LK with Tomasi’s features & pyramid.

CS4243

Motion Tracking

46

Appendix


Calculation of Ix, Iy, It
 Use

finite difference method
 Forward difference
Ix = I(x  1, y, t)  I(x, y, t)
Iy = I(x, y  1, t)  I(x, y, t)
It = I(x, y, t  1)  I(x, y, t)
 Backward difference
Ix = I(x, y, t)  I(x  1, y, t)
Iy = I(x, y, t)  I(x, y  1, t)
It = I(x, y, t)  I(x, y, t  1)

CS4243

Motion Tracking

47

Further Readings


Lucas-Kanade tracking with pyramid:
[BK08] Chapter 10.



Optical flow: [Sze10] Section 8.4.



Hierarchical motion estimation (with image
pyramid): [Sze10] Section 8.1.1.

CS4243

Motion Tracking

48

References


[Bir] S. Birchfield. KLT: An implementation of the Kanade-LucasTomasi feature tracker. http://vision.stanford.edu/~birch/klt/.



[BK08] Bradski and Kaehler. Learning OpenCV: Computer Vision
with the OpenCV Library. O’Reilly, 2008.



[LK81] B. D. Lucas and T. Kanade. An iterative image registration
technique with an application to stereo vision. In Proceedings of 7th
International Joint Conference on Artificial Intelligence, pages 674–
679, 1981.



[ST94] J. Shi and C. Tomasi. Good features to track. In Proceedings
of IEEE Conference on Computer Vision and Pattern Recognition,
pages 593–600, 1994.



[Sze10] R. Szeliski. Computer Vision: Algorithms and Applications.
Springer, 2010.

CS4243

Motion Tracking

49

References


CS4243

[TK91] C. Tomasi and T. Kanade. Detection and tracking of point
features. Technical Report CMU-CS-91-132, School of Computer
Science, Carnegie Mellon University, 1991.

Motion Tracking

50