Transcript Slide 1
Learning the “Epitome” of a Video Sequence Information Processing Workshop 2004 Vincent Cheung Probabilistic and Statistical Inference Group Electrical & Computer Engineering University of Toronto Toronto, Ontario, Canada Advisor: Dr. Brendan J. Frey Aug. 11, 2004 University of Toronto Information Processing Workshop 2004 Outline ● Image epitome ► ► What? Why? ● Implementation computation issues ► Efficiently implementing the learning algorithm ● Video epitome ► ► 1 / 12 Extension to videos Video inpainting Cheung Learning Image Information Processing Workshop 2004 Image Epitome ● Jojic, N., Frey, B., & Kannan, A. (2003). Epitomic analysis of appearance and shape. In Proc. IEEE ICCV. Video ● Miniature, condensed version of the image ● Accurately accounts for the interesting properties of the image ● Applications ► ► ► ► 2 / 12 object detection texture segmentation image retrieval compression Cheung Image Epitome Examples Video Learning Image Information Processing Workshop 2004 3 / 12 Cheung Learning the Image Epitome Learning Image Information Processing Workshop 2004 Training Set Video Input image Bayesian network T1 Unsupervised eLearning – epitome T2 Z1 4 / 12 Epitome e Sample Patches Z2 … TM Tk – mapping Zk – image patch ZM Cheung Shifted Cumulative Sum Algorithm Video Learning Image Information Processing Workshop 2004 (1, 1), (i, j) – row 1 + row (P+1) – col 1 + col (P+1) X Cumsum (2, 1), (i+1, j) 5 / 12 – N N + K e (1, 2), (i, j+1) 2 (i,j) K + + - - + pixel (1,1) + pixel (P+1, P+1) +N (2, 2), (i+1, N j+1) Cheung Collecting Sufficient Statistics Video Learning Image Information Processing Workshop 2004 X P e Ta P P Tß 6 / 12 P Cheung Video Learning Image Information Processing Workshop 2004 Extending Epitomes to Videos ● Desire a miniature, condensed version of a video sequence ● Want it to accurately account for the interesting properties of the video ● Applications ► ► ► ► ► ► ► 7 / 12 optic flow segmentation texture transfer layer separation compression noise reduction inpainting Cheung Video Epitome Learning Image Information Processing Workshop 2004 Training Set Video Input Video Sample Patches Unsupervised Learning Video Epitome Frame 3 Frame 2 Frame 1 8 / 12 Cheung Video Epitome Example Video Learning Image Information Processing Workshop 2004 Spatially Compressed Temporally Compressed 9 / 12 Cheung Learning Image Information Processing Workshop 2004 Video Inpainting (1) ● Fill in missing portions of a video ► ► damaged films occluding objects Video ● Reconstruct the missing pixels from the video epitome 10 / 12 Cheung Video Inpainting (2) Video Learning Image Information Processing Workshop 2004 11 / 12 Cheung Information Processing Workshop 2004 Conclusion ● Improved the efficiency of learning image epitomes ● Extended the concept of epitomes to video sequences ● Demonstrated the ability of video epitomes to model motion patterns through video inpainting 12 / 12 Cheung