Transcript Content-Preserving Warping

Single-view Video To Multi-view Video
台大資工系 黃昱翔, 黃彥翔
指導教授: 莊永裕老師
Goal
• 把2D所拍攝的靜物影片，經由建構出虛擬
cameras的方法，模擬人類雙眼視覺的效
果，再利用雙眼視差，經過處理，產生出
新的3D的影片。
Procedure
• Camera Path Tracking & Feature Points Detecting
• Virtual Camera Path Generating
• Feature Points Projecting
• Image Warping (frame by frame)
• Combine to a new 3D video
Camera Path Tracking & Feature Points
Detecting
• Structure from Motion~
• To figure out the camera path
切成一張張tga圖片
Camera Path Tracking & Feature Points
Detecting
• 經過parse這些output檔
可以得到camera位置以及其座標系、features的
3D位置、哪幾個frame存在哪些feature、相機焦
距等等資訊等等備用
Virtual Camera Path Generating
• 由取出的camera位置往其水平方向各延伸
出一單位長的距離建造 virtual cameras
Feature Points Projecting
• 再以以下公式將該frame所有feature points
投影到兩個virtual cameras
•
•
•
•
# [ x' ] = [f/sx
# [ y' ] = [0
# [ z' ] = [0
#
0
ppx] [ H0x
f/sy ppy] [ V0x
0
1 ] [ Ax
H0y
V0y
Ay
H0z ] [ 1 0 0 -Cx]
V0z ] [ 0 1 0 -Cy]
Az ] [ 0 0 1 -Cz]
[X]
[Y]
[Z]
[1]
• 得到兩張新的feature points 圖 以及與原圖
的對應關係
Content-Preserving Warps for 3D
Video Stabilization
Feng Liu
Michael Gleicher
Hailin Jin
Aseem Agarwala
Prerequisite
• 用影片拍靜物。
• 用 voodoo 找出 camera 路徑和feature位置。
• 在 camera 的左右邊各生出虛擬相機，當左右眼。
– 眼睛聚焦看 (stare)
– 眼睛平行看 (parallel)
• 把物體置中，紀錄 feature 點的位置。
– X Y XR YR XL YL
• Feature在原本相機的座標： X , Y
• Feature在右邊虛擬相機的座標： XR , YR
• Feature在左邊虛擬相機的座標： XL , YL
238.697
274.984
324.192
440.074
317.985
319.242
302.043
384.629
329.483
148.948
315.921
293.051
161.912
405.702
126.068
117.66
89.1555
70.5983
67.7392
196.682
59.8351
115.924
76.4908
53.4643
156.882
108.342
47.6016
130.359
155.048
192.123
244.748
364.159
243.452
241.479
229.323
298.994
252.881
105.444
235.337
210.254
117.539
317.013
126.068
117.66
89.1555
70.5983
67.7392
196.682
59.8351
115.924
76.4908
53.4643
156.882
108.342
47.6016
130.359
322.346
357.846
403.636
515.99
392.518
397.004
374.762
470.265
406.085
192.453
396.506
375.849
206.284
494.392
126.068
117.66
89.1555
70.5983
67.7392
196.682
59.8351
115.924
76.4908
53.4643
156.882
108.342
47.6016
130.359
Procedure
• Pre-warping
– Cut input image into square grids
– Geometry relation between grid and feature point
– Find homography
• Content-Preserving Warp
– Data term
– Similar transform term (smooth term)
– Optimization
• Texture Mapping
Cut input image into square grids
640 pixels
20 grids
Grid左上角的座標 (x, y)
480 pixels
32 x 32
Grid
15 grids
共 15 x 20 = 300 個格子，共 (15+1) x (20+1) = 336 個格子點。
Geometry relation between grid and
feature point
• 紀錄feature點在哪個grid裡面。
• 紀錄feature點在該grid裡面的位置關係。
W[0]
V1
L14
V4
W[3]
L12
W[1]
V2
P
V3
W[2]
Find Homography
• An initial guess
• Input:
– 原本圖片的feature點座標
– 左眼/右眼的feature點座標
• Output:
– A 3x3 Homography matrix
Find Homography
• 原本圖片的feature點，想要warp到左眼或
右眼相機裡feature點的位置。
homography
matrix
3x3
238.697
274.984
324.192
440.074
317.985
319.242
302.043
384.629
329.483
148.948
315.921
293.051
161.912
405.702
126.068
117.66
89.1555
70.5983
67.7392
196.682
59.8351
115.924
76.4908
53.4643
156.882
108.342
47.6016
130.359
#feature x 2
155.048
192.123
244.748
364.159
243.452
241.479
229.323
298.994
252.881
105.444
235.337
210.254
117.539
317.013
126.068
117.66
89.1555
70.5983
67.7392
196.682
59.8351
115.924
76.4908
53.4643
156.882
108.342
47.6016
130.359
#feature x 2
Result of Pre-warping
Procedure
• Pre-warping
– Cut input image into square grids
– Geometry relation between grid and feature point
– Find homography
• Content-Preserving Warp
– Data term
– Similar transform term (smooth term)
– Optimization
• Texture Mapping
Data Term
• 主要限定格子點位置的term，定義格子點
與feature之關係
– 所屬grid的四周頂點座標 (Vk)
– 在grid裡面的位置關係 (wkT)
– 新的feature在grid裡，位置(wk)應和原本差不多
Pk 是左眼或右眼feature點的位置
填矩陣
(Initial
Guess改
過的)
新的
格子點 Feature
座標
座標
20個格子 (21個格子點)
15個格子
(16個格子
點)
2 x 21 x 16 = 672
2xn
A
x
b
Similar Transform Term
• Warp之後，保持格子點之間的相互位置關
係不要變化太多而造成扭曲。
– 找出原本格子點V1, V2, V3的相互位置關係
– 新的格子點V1’, V2’, V3’希望也具有這樣的關係
Ws
= Salience of the grid
= Var(intensity of pixels in the grid) + 0.5
V1
V2
V3
找格子點相互位置關係 (找u,v)
V2
u
V1
V3
V2
V3
V1
V3
V2
V3
V2
v
V2
V3
u, v 即格子點之間的相互位置關係
同理,
新的格子點也要符合u, v
* 每個點要把所有鄰居的constrain加起來，以下只包含一個constrain。


2 x 21 x 16 = 672
672(每個點輪流當V1)
A
x
b
Optimization
• E  E d  E s
– α: smoothness weight
– A linear system in Ax=b form
• can be easily solved by using SVD
Optimization
2 x 21 x 16 = 672
2xn
672
A
 解出的x矩陣就是新的格子點座標。
x
b
Procedure
• Pre-warping
– Cut input image into square grids
– Geometry relation between grid and feature point
– Find homography
• Content-Preserving Warp
– Data term
– Similar transform term (smooth term)
– Optimization
• Texture Mapping
Texture Mapping
• Input:
– 新格子點座標(上一步得到的x矩陣)
– 存成OpenGL texture的原圖
• 一格一格貼
Combine to a new 3D video
• demo