Transcript Alpha Matting - The BSOE Alumni Server

Background substitution
Che-An Wu
Objective
•Extract the foreground object and put into another background
Depth
Map
Trimap
AlphaMap
Background
Substitution
Environment
 Platform
 Laptop vs Kinect

in Java with library OpenNI and Processing
 Assumption
 fixed camera position
 the interest parts in the video are human
 human in the scene are closer than background
Extract the
foreground object Depth information from Kinect
 Depth color picture
 By the method SimpleOpenNI.depthImage()
 Black : pixels without depth measurement
Extract the
foreground object Generate a trimap
 Trimap
 User-generated mask map
 made up of three zones



Black : Background
White : Foreground
Gray : Unknown
Extract the
foreground object Generate a trimap
 Trimap (right)


 White : Foreground (d ≤ 𝑑𝑡ℎ𝑟𝑒𝑠ℎ )
 Black : Background (d > 𝑑𝑡ℎ𝑟𝑒𝑠ℎ )
 Grey : Unknown
(pixels with no depth measurement)
Extract the
foreground object Generate a trimap
 Trimap


 White : Foreground (d ≤ 𝑑𝑡ℎ𝑟𝑒𝑠ℎ )
 Black : Background (d > 𝑑𝑡ℎ𝑟𝑒𝑠ℎ )
 Grey : Unknown
(pixels with no depth measurement)
Extract the
foreground object Develop algorithm to determine Unknown area
 Alpha matte (Compositing Equation)
𝐶𝑝 = 𝛼𝑝 𝐹𝑝 + (1 − 𝛼𝑝 )𝐵𝑝

𝐶𝑝 ∶ 𝑜𝑏𝑠𝑒𝑟𝑣𝑒𝑑 𝑐𝑜𝑙𝑜𝑟 𝑜𝑓 𝑝𝑖𝑥𝑒𝑙

𝐹𝑝 ∶ 𝑓𝑜𝑟𝑒𝑔𝑟𝑜𝑢𝑛𝑑 𝑐𝑜𝑙𝑜𝑟 𝑜𝑓 𝑝𝑖𝑥𝑒𝑙

𝐵𝑝 ∶ 𝑏𝑎𝑐𝑘𝑔𝑟𝑜𝑢𝑛𝑑 𝑐𝑜𝑙𝑜𝑟 𝑜𝑓 𝑝𝑖𝑥𝑒𝑙

𝛼𝑝 ∶ 𝑜𝑝𝑎𝑐𝑖𝑡𝑦 𝑣𝑎𝑙𝑢𝑒, 𝑖𝑛𝑡𝑒𝑟𝑝𝑜𝑙𝑎𝑡𝑖𝑜𝑛 𝑝𝑎𝑟𝑎𝑚𝑒𝑡𝑒𝑟
Extract the
foreground object Cross Bilaterial Filter[1]
 Alpha matte - Algorithm
 𝜶𝒋 is the alpha-value from the sparse alpha-matte
 f is the spatial filter kernel (in our case, a Gaussian




centered at i)
g is the range filter kernel (also a Gaussian)
I is the color image
N is the neighborhood surrounding I (square window)
K is a normalizing factor
Extract the
foreground object Cross Bilaterial Filter[1]
 Alpha matte – Result (right)
 The unknown area color value is been determined by
substituting the alpha value estimated from the algorithm
into the compositing equation 𝐶𝑝 = 𝛼𝑝 𝐹𝑝 + (1 − 𝛼𝑝 )𝐵𝑝
Che-An Wu
Putting into another
background
Determine the color of new image
 Known
 Alpha value of unknown area
 New background image
 Foreground value from original scene

𝐶𝑝 = 𝛼𝑝 𝐹𝑝 + (1 − 𝛼𝑝 )𝐵𝑝
Putting into another
background
Determine the color of new image
background substitution
Alpha map
Putting into another
background
Results
 Real-time parameters setting
 Depth threshold = 2m
 Neighborhood window size = 13*13;
 Spatial Filter Gaussian sigma = 13*13*2%
 Range Filter Gaussian sigma 30
 Improvement
 Adjusting the neighborhood window definition
=>

or
blue area is the pixels in the neighborhood window
 The result is almost the same but we can adjust the
parameter of window size twice to be 19*19
Putting into another
background
Problem (Ongoing)
 The FOV of the rgb camera and depth camera
(combine to camera in order to generate depth image)
is slightly different
 It still need to calibrate both camera in order to have
the good result of background substitution.
 The result C=aF+(1-a)B the alpha location cannot
match the foreground location.
References
 [1] Crabb, Ryan, et al. "Real-time foreground
segmentation via range and color imaging.“ Computer
Vision and Pattern Recognition Workshops, 2008.
CVPRW'08. IEEE Computer Society Conference on.
IEEE, 2008.
Che-An Wu