Transcript Video Epitome - VincentCheung.ca

Modelling Motion Patterns with Video
Epitomes
Machine Learning Group Meeting
University of Toronto
Oct. 18, 2004
Vincent Cheung
Probabilistic and Statistical Inference Group
Electrical & Computer Engineering
University of Toronto
Toronto, Ontario, Canada
Advisor: Dr. Brendan J. Frey
Oct. 18, 2004
University
of Toronto
ML Group Meeting, Oct. 18, 2004
Outline
● Image epitome
►
►
►
What?
Why?
How?
● Implementation computation issues
►
Efficiently implementing the learning algorithm
● Video epitome
►
►
1 / 18
Extension to videos
Filling-in missing information
Cheung
Learning
Image
ML Group Meeting, Oct. 18, 2004
Image Epitome
● Jojic, N., Frey, B., & Kannan, A. (2003). Epitomic
analysis of appearance and shape. In Proc. IEEE
ICCV.
Video
● Miniature, condensed version of the image
● Models the image’s shape and textural
components.
● Applications
►
►
►
►
2 / 18
object detection
texture segmentation
image retrieval
compression
Cheung
Image Epitome Examples
Video
Learning
Image
ML Group Meeting, Oct. 18, 2004
3 / 18
Cheung
Learning the Image Epitome (1)
Video
Learning
Image
ML Group Meeting, Oct. 18, 2004
4 / 18
Cheung
Learning the Image Epitome (2)
Learning
Image
ML Group Meeting, Oct. 18, 2004
Training Set
Video
Input image
Bayesian
E:
network
Epitome
e
Sample
Patches
T1
Unsupervised
eLearning
– epitome
T2
Z1
Z2
…
TM
Tk – mapping
Zk – image patch
ZM
M:
5 / 18
Cheung
Shifted Cumulative Sum Algorithm
Video
Learning
Image
ML Group Meeting, Oct. 18, 2004
(1, 1), (i, j)
– row 1
+ row (P+1)
– col 1
+ col (P+1)
X
Cumsum
(2, 1), (i+1, j)
6 / 18
–
N
N
+
K
e (1, 2), (i, j+1)
2
(i,j)
K
+
+
-
-
+ pixel (1,1)
+ pixel
(P+1, P+1)
+N
(2, 2), (i+1,
N j+1)
Cheung
Collecting Sufficient Statistics
Video
Learning
Image
ML Group Meeting, Oct. 18, 2004
X
P
e
Ta
P
P
Tß
7 / 18
P
Cheung
Video
Learning
Image
ML Group Meeting, Oct. 18, 2004
Extending Epitomes to Videos
● Desire a miniature, condensed version of a video
sequence
● Want the epitome to model the basic shape,
textural, and motion patterns of the video
● Applications
►
►
►
►
►
►
►
8 / 18
optic flow
segmentation
texture transfer
layer separation
compression
noise reduction
fill-in / inpainting
Cheung
Video Epitome
Learning
Image
ML Group Meeting, Oct. 18, 2004
Training Set
Video
Input Video
Sample
Patches
Unsupervised
Learning
Video
Epitome
Frame 3
Frame 2
Frame 1
9 / 18
Cheung
Video Epitome Example
Video
Learning
Image
ML Group Meeting, Oct. 18, 2004
Spatially Compressed
Temporally Compressed
10 / 18
Cheung
Learning
Image
ML Group Meeting, Oct. 18, 2004
Video Inpainting (1)
● Fill in missing portions of a video
►
►
damaged films
occluding objects
Video
● Reconstruct the missing pixels from the video epitome
11 / 18
Cheung
Video Inpainting (2)
Video
Learning
Image
ML Group Meeting, Oct. 18, 2004
12 / 18
Cheung
Learning
Image
ML Group Meeting, Oct. 18, 2004
Filling-in Missing Data
Likelihood
Video
Joint
Variational
Approx
E-Step
M-Step
13 / 18
Cheung
Learning
Image
ML Group Meeting, Oct. 18, 2004
Missing Channels
● Generalization of the video inpainting problem
● Inpainting
Video
►
Missing entire pixels
● Missing Channels
►
Missing one or more of the red, green, or blue (RGB)
components of a given pixel
● Epitome must consolidate multiple patches
together to piece together the missing channel
information
►
►
14 / 18
No training patch contains all the channel information
Use the epitome to fill-in the missing data
Cheung
Image Missing Channels Fill-in
Video
Learning
Image
ML Group Meeting, Oct. 18, 2004
15 / 18
Cheung
Video Missing Channels Fill-in
Video
Learning
Image
ML Group Meeting, Oct. 18, 2004
16 / 18
Cheung
ML Group Meeting, Oct. 18, 2004
Conclusion
● Improved the efficiency of learning image
epitomes
● Extended the concept of epitomes to video
sequences
● Demonstrated the ability of video epitomes to
model motion patterns through video inpainting
17 / 18
Cheung
ML Group Meeting, Oct. 18, 2004
Future Work
● Determining the size of the epitome
►
Dependent on the complexity of the image / video
■ Minimum description length
■ Variational Bayesian
● Optimal patch size(s)
►
Problem specific
● Additional transformations into the epitome
►
►
Rotation
Scale
● Additional video epitome applications
►
►
►
18 / 18
Super-resolution
Layer separation
Object recognition
Cheung