Transcript 슬라이드 1
Webcam-synopsis: Peeking Around the World Young Ki Baik (CV Lab.) 2008. 4. 4 (Fri) Webcam-Synopsis: Peeking Around the World References Webcam Synopsis : Peeking Around the World • Yael Pritch, Alex Rav-Acha, Avital Gutman Shmuel Peleg • (ICCV 2007) Non-Chronological Video Synopsis and Indexing • Yael Pritch, Alex Rav-Acha, Shmuel Peleg (PAMI 2008) Background Cut • J. Sun, W. Zhang, X. Tang, and H. Shum (ECCV 2006) Webcam-Synopsis: Peeking Around the World What is Video-Synopsis? Video synopsis is compact video data reducing spatio-temporal redundancy in input video. Webcam-Synopsis: Peeking Around the World Demo (Final result) Webcam-Synopsis: Peeking Around the World How can we make Video Synopsis? Basic concept • Detect activity tubes… in input video I. • Put activity tubes… to the S… considering relation between - tubes and background. - tube and other tubes. Webcam-Synopsis: Peeking Around the World Synopsis Properties of video synopsis S • The video synopsis S should be substantially shorter than the original video I. • Maximum “activity” from the original video should appear in the synopsis video. • The dynamics of the objects should be preserved in the synopsis video. (for example, regular fast-forward may fail to preserve the dynamics of fast objects.) • Visible seams and fragmented objects should be avoided. Webcam-Synopsis: Peeking Around the World OBJECT-BASED SYNOPSIS Webcam-Synopsis: Peeking Around the World Object-based synopsis construction Online Phase • This phase is done in real time during video capture. • Preparing step for response phase. Response Phase • started with user query. • This phase may take a few minutes, depending on the activity in the time period of interest. Webcam-Synopsis: Peeking Around the World Overall flow Webcam-Synopsis: Peeking Around the World ONLINE PHASE Webcam-Synopsis: Peeking Around the World Video Acquisition Video Volume Data t y I(x, y, t) (x, y) : t : : 3D Space Time Volume Spatial coordinates of the image Frame index x Webcam-Synopsis: Peeking Around the World Background Video Construction Temporal Median t Temporal Window y Sorting by intensity Selecting x median value 4 minutes Webcam-Synopsis: Peeking Around the World Background Video Construction t t y y x x I B Webcam-Synopsis: Peeking Around the World Moving Objects(tubes) Extraction Using Background Cut (ECCV 2006) • Using min-cut to get a smooth segmentation of foreground objects. • Only considering 2D image (not video). Webcam-Synopsis: Peeking Around the World Moving Objects(tubes) Extraction Extracting moving objects • We have information for… I(x, y, t) : 3D Space Time Volume B(x, y, t) : 3D Space Time Volume for background Current ~ I : Current image = I (x, y, current time) B : Current background image = B (x, y, current time) Webcam-Synopsis: Peeking Around the World Moving Objects(tubes) Extraction Simplify Object detection problem to… • Binary labeling f : Foreground = 1, Background = 0 • Obtaining the desirable labeling f … → with Gibbs energy function. E( f ) E1 f r rI Color term r ,s Neighbors E2 f r , f s Contrast term r : target position of image s : neighboring of target position (4 Neighbors) : user defined weight Webcam-Synopsis: Peeking Around the World Moving Objects(tubes) Extraction Extracting moving objects • Unary term (or color term) E( f ) E1 f r rI r ,s Neighbors d I r B r E2 f r , f s Color differences between image and background Foreground energy 0 E1 1 k1 d Background energy d k1 otherwise k1, k2 : user defined thresholds E1 0 d - k1 0 d k2 k2 d k1 otherwise Webcam-Synopsis: Peeking Around the World Moving Objects(tubes) Extraction Using Background Cut (ECCV 2006) • Using min-cut to get a smooth segmentation of foreground objects. • Only considering 2D image (not video). Webcam-Synopsis: Peeking Around the World Moving Objects(tubes) Extraction Extracting moving objects • Binary term (or contrast term) E( f ) E1 f r rI r ,s Neighbors E2 f r , f s E2 ( f r , f s ) f r f s Exp I r I s In case of same labeling, energy is decreased. In case of different labeling, energy defends on differences between neighboring intensities. : user defined weight 2 Webcam-Synopsis: Peeking Around the World Moving Objects(tubes) Extraction Using min-cut algorithm… - Moving objects are extracted. Webcam-Synopsis: Peeking Around the World Moving Objects(tubes) Extraction Extracting moving objects • Constructing a mask of all foreground pixels in space-time volume. • Applying a 3D morphological dilation on mask. • Finally we can obtain activity tubes (or objects). t x Example of tubes Webcam-Synopsis: Peeking Around the World Moving Objects(tubes) Extraction Object queue • Obtained activity tubes are saved in queue. End of online phase… Webcam-Synopsis: Peeking Around the World RESPONSE PHASE Webcam-Synopsis: Peeking Around the World Response phase User query When user query occurred, response phase are started. • “I would like to watch in one minute a synopsis of the video from this camera captured during the last hour.” • “I would like to watch in five minutes a synopsis of the last week.”… Input video Synopsis Webcam-Synopsis: Peeking Around the World Response phase User query • “I would like to watch in one minute a synopsis of the video from this camera captured during the last hour.” • “I would like to watch in five minutes a synopsis of the last week.”… Webcam-Synopsis: Peeking Around the World Creating time lapse background video Time lapse background video (Bout) • The background of the synopsis video • It should represent the background changes over time. • Day-night transitions, etc. • It should represent the background of the activity tubes. Background Video Sampling uniformly Time lapse BG video Webcam-Synopsis: Peeking Around the World Creating time lapse background video Time lapse background video • A temporal activity histogram Ha • A uniform temporal histogram Ht Webcam-Synopsis: Peeking Around the World Creating time lapse background video Time lapse background video • A temporal activity histogram Ha • A uniform temporal histogram Ht • Interpolating the two histograms Hi = Ha + (1- λ)Ht Background Video (b) Sampling with Hi Bout Time lapse BG video Webcam-Synopsis: Peeking Around the World Selecting tubes and stitching Definition M b bˆ M : temporal mapping (or time shift) s e t t b : tube with time segment b b , tb ˆb : shifted tube with time segment tˆb tˆbs , tˆbe Input video b M bˆ Synopsis Webcam-Synopsis: Peeking Around the World Select tubes and stitching Energy for temporal mapping M M b bˆ E bˆ, bˆ E bˆ, bˆ E M bQ Ea bˆ Es bˆ b ,bQ Q , , t c : Queue b : target tube : user selected weights b : other tube Webcam-Synopsis: Peeking Around the World Select tubes and stitching Graph Node number = number of tube Longest tube tube tube Synopsis tube t tube label label Label = number of frame t label label Webcam-Synopsis: Peeking Around the World Select tubes and stitching Energy for temporal mapping M • Unary term bQ E bˆ E bˆ a s Synopsis • Activity Cost 0 Ea bˆ = ˆ I x, y , t B x , y , t x,y,tb tube region inside the S tube region outside the S Only pixel that were not entered into the synopsis are added to the activity cost. Webcam-Synopsis: Peeking Around the World Select tubes and stitching Energy for temporal mapping M • Unary term bQ E bˆ E bˆ a s Synopsis • Consistency with background Es bˆ = x,y bˆ ,ttbˆ tout bˆ : I bˆ x, y, t Bout x, y, t border of the mapped activity tube Webcam-Synopsis: Peeking Around the World Select tubes and stitching Energy for temporal mapping M • Binary term b,bQ Et bˆ, bˆ Ec bˆ, bˆ • Collision Cost Ec bˆ = x,y,tˆt b ˆt b I x, y, t B x, y, t bˆ I x, y, t B x, y, t bˆ This expression give a low penalty to pixel whose color is similar to the background… Synopsis Webcam-Synopsis: Peeking Around the World Select tubes and stitching Energy for temporal mapping M • Binary term b,bQ Et bˆ, bˆ Ec bˆ, bˆ • Temporal Consistency Cost Et bˆ Preserving the chronological order of events Synopsis Webcam-Synopsis: Peeking Around the World Select tubes and stitching E(M) is minimized by Min-cut algorithm. Stitching the synopsis video • To make more reliable result… → The α–Poisson Image Blending End of response phase… Webcam-Synopsis: Peeking Around the World Conclusion The method to creating a short video that is a synopsis of and video stream has been presented. Discussion Contribution • Found out new interesting application. Can we find more contribution? • Finding higher dimensional problem and Solving it. • 3D volume animation, etc.