Transcript week08-motionSlides.ppt

Computing Motion from Images
Chapter 9 of S&S plus otherwork.
Stockman MSU Fall 2008
1
General topics






Low level change detection
Region tracking or matching over time
Interpretation of motion
MPEG compression
Interpretation of scene changes in
video
Understanding human activites
Stockman MSU Fall 2008
2
Motion important to human vision
Stockman MSU Fall 2008
3
What’s moving: different cases
Stockman MSU Fall 2008
4
Image subtraction
Simple method to remove
unchanging background from
moving regions.
Stockman MSU Fall 2008
5
Change detection for surveillance
Stockman MSU Fall 2008
6
Change detection by image subtraction
Stockman MSU Fall 2008
7
What to do with regions of change?




Discard small regions
Discard regions of non interesting
features
Keep track of regions with interesting
features
Track in future frames from motion plus
component features
Stockman MSU Fall 2008
8
Some effects of camera motion
that can cause problems
Stockman MSU Fall 2008
9
Motion field
Stockman MSU Fall 2008
10
FOE and FOC
Will return to use the FOE or FOC or detection of panning
to determine what the camera is doing in video tapes.
Stockman MSU Fall 2008
11
Gaming using a camera to
recognize the player’s motion
Decathlete game
Stockman MSU Fall 2008
12
Decathlete game
Cheap camera
replaces usual
mouse for input
Running speed and jumping of the avatar is controlled by
detected motion of the player’s hands.
Stockman MSU Fall 2008
13
Motion detection input device
Running (hands)
Jumping (hands)
Stockman MSU Fall 2008
14
Motion analysis controls
hurdling event (console)
•Top left shows video frame of player
•Middle left shows motion vectors from multiple frames
•Center shows jumping patterns
Stockman MSU Fall 2008
15
Related work





Motion sensed by crude cameras
Person dances/gestures in space
System maps movement into music
Creative environment?
Good exercise room?
Stockman MSU Fall 2008
16
Computing motion vectors
from corresponding “points”
High energy neighborhoods are
used to define points for matching
Stockman MSU Fall 2008
17
Match points between frames
Such large
motions are
unusual. Most
systems track
small motions.
Stockman MSU Fall 2008
18
Requirements for interest points
Match small neighborhood to small neighborhood. The
previous “scene” contains several highly textured
neighborhoods.
Stockman MSU Fall 2008
19
Interest = minimum
directional variance
Used by Hans Moravec in his robot stereo vision system.
Interest points were used for stereo matching.
Stockman MSU Fall 2008
20
Detecting interest points in I1
Stockman MSU Fall 2008
21
Match points from I1 in I2
Stockman MSU Fall 2008
22
Search for best match of point
P1 in nearby window of I2
For both motion and stereo, we have some constraints on where
to search for a matching interest point.
Stockman MSU Fall 2008
23
Motion vectors clustered to
show 3 coherent regions
Motion coherence:
points of same
object tend to
move in the same
way
All motion vectors are
clustered into 3 groups
of similar vectors
showing motion of 3
independent objects.
(Dina Eldin)
Stockman MSU Fall 2008
24
Two frames of aerial imagery
Video frame N and N+1 shows slight movement:
most pixels are same, just in different locations.
Stockman MSU Fall 2008
25
Can code frame N+d with
displacments relative to frame N





for each 16 x 16 block in the 2nd image
find a closely matching block in the 1st
image
replace the 16x16 intensities by the
location in the 1st image (dX, dY)
256 bytes replaced by 2 bytes!
(If blocks differ too much, encode the
differences to be added.)
Stockman MSU Fall 2008
26
Frame approximation
Left is original video frame N+1. Right is set of best image
blocks taken from frame N. (Work of Dina Eldin)
Stockman MSU Fall 2008
27
Best matching blocks between video
frames N+1 to N (motion vectors)
The bulk of the vectors
show the true motion of
the airplane taking the
pictures. The long vectors
are incorrect motion
vectors, but they do work
well for compression of
image I2!
Best matches from 2nd to first image shown as vectors
overlaid on the 2nd image. (Work by Dina Eldin.)
Stockman MSU Fall 2008
28
Motion coherence provides
redundancy for compression
MPEG “motion compensation”
represents motion of 16x16 pixels
blocks, NOT objects
Stockman MSU Fall 2008
29
MPEG represents blocks that
move by the motion vector
Stockman MSU Fall 2008
30
MPEG has ‘I’, ‘P’, and ‘B’ frames
Stockman MSU Fall 2008
31
Computing Image Flow
Stockman MSU Fall 2008
32
Stockman MSU Fall 2008
33
Assumptions
Stockman MSU Fall 2008
34
IMAGE
FLOW
EQUATION
1 of 2
Stockman MSU Fall 2008
35
Image flow equation 2 of 2
Stockman MSU Fall 2008
36
Aperture problem
Stockman MSU Fall 2008
37
Solving flow by propagation of
constraints
Stockman MSU Fall 2008
38
Info at corner constrains the
flow along both edges
Solve constraints
using contraint
propagation or
differential equation
with boundary
conditions.
Stockman MSU Fall 2008
39
Tracking several objects
Use assumptions of physics to
compute multiple smooth paths.
(work of Sethi and R. Jain)
Stockman MSU Fall 2008
40
Stockman MSU Fall 2008
41
Tracking in images over time
Stockman MSU Fall 2008
42
General constraints from physics
Stockman MSU Fall 2008
43
Other possible constraints




Background statistics stable
Object color/texture/shape might
change slowly over frames
Might have knowledge of objects under
survielance
Objects appear/disappear at boundary
of the frame
Stockman MSU Fall 2008
44
Stockman MSU Fall 2008
45
Sethi-Jain algorithm
Stockman MSU Fall 2008
46
Stockman MSU Fall 2008
47
Total smoothness of m paths
Stockman MSU Fall 2008
48
Greedy exchange algorithm
Stockman MSU Fall 2008
49
Example data structure
Total smoothness for trajectories of Figure 9.14
Stockman MSU Fall 2008
50
Example of domain specific tracking (Vera Bakic)
Tracking eyes and
nose of PC user.
System presents
menu (top). User
moves face to
position cursor to a
particular box
(choice). System
tracks face
movement and
moves cursor
accordingly: user
gets into feedbackcontrol loop.
Stockman MSU Fall 2008
51
Segmentation of videos/movies
Segment into scenes, shots,
specific actions, etc.
Stockman MSU Fall 2008
52
Types of changes in videos
Stockman MSU Fall 2008
53
How do we
compute the
scene change?
Anchor person scene
at left
Scene break
Street scene for
news story
Stockman MSU Fall 2008
From Zhang et al
1993
54
Histograms of frames across
the scene change
Histograms at left are from
anchor person frames, while
histogram at bottom right is
from the street frame.
Stockman MSU Fall 2008
55
Heuristics for ignoring zooms
Stockman MSU Fall 2008
56
American sign language
example
Stockman MSU Fall 2008
57
Example from Yang and Ahuja
Stockman MSU Fall 2008
58
Stockman MSU Fall 2008
59
Stockman MSU Fall 2008
60